2.1. The Cryptography of the J-19 System
As mentioned in the previous chapter, like many countries, Japan relied on a mix of ciphers and codes to secure its diplomatic traffic. Also, Japan was one of the earliest countries to rely on cipher machines to encrypt some of its more sensitive or important communications. However, these machines were expensive to make, maintain, and protect. In the summer of 1940, only about ten diplomatic missions held Red and Purple cipher machines, though more were scheduled to receive the machines.[] Also, these devices could be available only at missions whose physical facilities were considered secure from physical compromise to the intelligence services of host countries. The physical security status of Japan's overseas diplomatic facilities was heavy on the minds of Tokyo's diplomats. In April and December 1940, the Foreign Ministry queried its overseas facilities about physical security. The results might not have encouraged Tokyo about security at many of the sites such as the embassy in Bogota, Colombia, and the consulate in New Orleans in the United States where the sensitive material was stored in rooms below the consul's bedroom with no sentry save the diplomat sleeping upstairs.[] In late December 1940, Tokyo informed its embassy in Washington that it was shipping a metal safe that was about two meters high, a little over a meter wide and one meter deep. In this safe would go all cryptographic material, including the embassy's HINOKI (Purple) machines and all manual codes and ciphers.[] However, the security measures for Washington would not work at most of the other facilities because of a lack of space and insufficient Japanese personnel to maintain security.
The problem for Tokyo in 1940 was that if an important or top-secret message needed to be sent to all stations – a "circular" – encrypting it with either the Purple or Red machine was no solution. The vast majority of Japanese missions did not have either device. A further cryptographic system was needed to supplement the machine systems; yet it had to be secure to the point where sensitive and secret traffic to all diplomatic sites could be encrypted by it. The solution was to field a new manual code with a particularly complex encryption system.
2.1.1. Japan Fields a New Diplomatic Manual Cryptographic System
In mid-June 1940, the Japanese Foreign Office informed all of its overseas stations that a new supplementary manual cryptographic system would soon be put into effect and that they would receive the appropriate code books and auxiliary systems associated with it.[] The system was called MATSU (Pine). It consisted of two code tables with digraph and tetragraph values for Japanese Kana (phonetic Japanese syllabary) along with two auxiliary encryption systems, a substitution table and a transposition cipher, designated K-5, by which resulting code text messages would be encrypted for greater security. Both the tables and the auxiliary transposition encryption system were substantial advances in size and complexity over the immediately preceding diplomatic cryptographic systems, notably J-11 through J-15.
About every three months thereafter, this new manual code and cipher system would undergo major upgrade, that is, a new code table and auxiliary system would be introduced, with a total of three changes completed through mid-1941. MATSU was labeled J-16 by the Americans. The final system in this series progression would be designated J-19 by the Americans. The J-19 system would be an important part of the Winds controversy. At the same time, the story of its changes and the American solutions reveals much of the situation of American cryptanalysis against systems other than Purple.
Below is a list of the system designators, reading from the left, the American designator, the auxiliary transposition cipher designator ("K"), the Japanese covername and the effective dates:
Figure 2.2. MATSU code successor systems
The code tables for MATSU were designated J-16 by the Americans soon after they recognized the initial intercepted messages were encoded in groups substantially different from the current code, J-15. MATSU and its successors were considerably more complex than the systems that preceded them. MATSU, with its code structure and the auxiliary systems, in fact, was a quantum leap in size, scope, and sophistication over the previous manual systems used by Tokyo's diplomats. The MATSU code charts were twice as large as those tables for the immediate predecessors J-11 through J-15. These earlier systems, which were in effect from 1939 through mid-1940, carried about 400 to 800 total code group entries consisting of plain text syllables with the corresponding digraphs, tetragraphs, and even the occasional trigraph (three-letter) code group. MATSU carried nearly double the number of code group entries – a total of almost 1,600 code groups. In the final version of MATSU, the J-19 system (with the Japanese cover name of FUJI), for example, the digraph LW represented the kana syllable SHI, and the two-letter group KP represented the syllable HA, and so on. These digraphs formed one code table that contained 676 code-for-text values. The American code-breakers would reconstruct this table in an analog fashion by creating a decrypt chart, being far easier to use for the decryption of messages. [See Exhibit #1][]
Punctuation and format requirements in messages, such as periods, commas, parentheses, line feeds, and new paragraphs were represented with separate and specific two-letter code groups. For example, the digraph "NC" corresponded to the start of a new paragraph in the message.
An accompanying second code chart contained four-letter code groups, which were used for items of text that were too difficult or clumsy to encode by syllable or letter substitution from the two-letter chart. These items included common foreign words, usually of a technical nature, proper names, geographic locations, months of the year, etc. There were 900 such four-letter code-for-text values on this second chart. These code groups were nested in the regular two-letter code text, segregated in the text by the two-letter code groups for special characters, such as HL for the open "[" (open brackets) character. This latter digraph was one of the special two-letter code groups used to alert the Japanese code clerk who was either decoding or encoding the message to refer to the chart of four-letter code groups.
There were two auxiliary cipher systems that were to be used to encrypt the coded messages: the Q-1 substitution system and the K-5 transposition system. The Q-1 process involved a complicated process of adding randomly selected letters to single letters in a coded message text, which were then replaced, or substituted with random two-letter groups from either of two substitution tables. Either of two five-letter indicators – CIFOL and VEVAZ – would appear as the first group of a message and pointed to one of two deciphering tables for the code clerk to use. However, this complicated auxiliary system seems to have been used only rarely.[]
The auxiliary transposition system, designated K-5, was used almost always to encrypt a message encoded with J-16 and its successors. At its basic level, a transposition cipher mixes the order of the elements of a message's text, whether plaintext or coded text. Generally, the plaintext or coded/cipher text is inscribed horizontally into a matrix, or "cage," of columns and rows of a dimension specified by the length of the message. The cipher text is created when the text is transcribed vertically (or "read out") from the columns of the cage in a specified order, which is established by a "key." A simplified version is given in the example below:
Step 1. The message plaintext, TOMORROW ALL UNITS ATTACK THE TOWN AT DAWN, is inscribed in a 6×6 matrix horizontally:
| 1 | 2 | 3 | 4 | 5 | 6 |
| T | O | M | O | R | R |
| O | W | A | L | L | U |
| N | I | T | S | A | T |
| T | A | C | K | T | H |
| E | T | O | W | N | A |
| T | D | A | W | N | -- |
Step 2. The "key" for this message is 3-1-6-2-5-4. The text is now transcribed (or "read out") vertically in the order set by the key.
The transposed text now reads (The cipher text is arranged in groups of five for easier transmission. Extra letters or spaces were often filled with "dummy" characters to achieve a complete 5-letter group.):
MATCO ATONT ETRUT HAOWI ATDRL ATNNO LSKWW
To decipher the message, the recipient reverses the process and inscribes the enciphered text into the matrix in the order set by the "key" and then reads out the text horizontally.
The K-5 transposition system, and all of its follow-on systems, naturally was far more sophisticated than the above example. In fact, the K-5 system was a major advance over previous transposition systems. Earlier ones that were implemented in 1939, notably K-1 through K-4, used an unfilled matrix for transcribing the coded text. Matrices or cages varied in width, namely, the number of vertical columns, from six to fourteen. The systems also used sets of five keys for transcribing text out of the matrix that were effective for a month. These keys were used on designated days in the month, usually six days picked at random or in a specified sequence of days, such as 1-6-11-16-21-26-31, 2-7-12-17-22-27, etc.
The K-5 transposition system was a major step forward for Japanese diplomatic cryptography because of two innovations. The first was that the parameters for the matrix into which the code groups were inscribed had advanced significantly in both complexity and size. This more sophisticated matrix was the hallmark characteristic of the new transposition system and appears to have been the foundation for the K-5 auxiliary system and all of its successors through K-10. The second innovation involved the use of more frequently changed key settings for the transposition sequence. The K-5 system used one hundred keys for three months as a daily-changing key. The key also varied more in length, being as long as twenty-five elements instead of the old maximum of fourteen. As the Japanese superseded the basic code charts in J-16, the associated transposition system simultaneously would be replaced.
The K-10 auxiliary transposition system associated with the J-19 code merits a detailed description. The K-10 cipher operated in continuous, nonrepeating, ten-day periods for each month, divided by the numbered days of the month 1–10, 11–20, and day 21 to the end of the month. Each ten-day period used a separate transposition matrix or cage, which the American codebreakers would refer to as a "form." A form was a cage at its largest twenty-five columns wide (horizontally) and up to thirty-five rows deep (vertical). The significant aspect of the form was the presence of randomly placed filled spaces in the cage, actually called "blanks," which resembled the nulls in a crossword puzzle. The appearance of blanks had the effect of breaking the flow of the coded text and created irregular lengths for each column of text. This was intended to make the system more secure through irregular segments of the complete text. The Japanese numbered the forms sequentially and each had a unique arrangement of blanks. The Americans called this type of form with embedded blanks a "stencil." Exhibit #2][] The Americans designed the blanks for their stencils used for decrypting J-19 messages either as inscribed dark spaces or with punch-outs, much like a grill.
As for the key, in K-10 system, the Japanese used a daily-changing key for the stencil. This key, which defined the number of columns used in a form, was a string of digits from 1 to 25, which varied in length daily from nineteen to twenty-five positions. The predecessors of the K-10 used different sets of key, or "banks." The first one, K-5, associated with J-16, used one hundred separate keys with lengths from fifteen to twenty-five positions. The next variant, K-6 (associated with J-17), merely reused the same one hundred keys, though with different indicators, that is, code groups that pointed to the key to be used for that period. The K-10 key took the process a further step by increasing by nearly a factor of four the possible key values. It is quite possible that the Japanese had generated all possible key settings for all possible key lengths and then randomly selected a number of them for use. By mid-1942, the Japanese may have exhausted this key library, for they began to reuse old 1941 key for messages encoded in J-19. However, they did not just reuse old key, but devised a method for relocating elements in the key string according an algorithm 1-3-5-7...10-8-6-4-2. A second method was devised in which old key strips were added "falsely" together, that is, dropping the resulting tens-position digit.[]
As mentioned above, the forms were potentially twenty-five columns wide, but a shorter key defined a "thinner" form, diminished from the right-hand side sliding to the left. The final width of the ten-day form was determined daily by the length of that day's key. The height or depth of the form was determined by the length of the message being encrypted: the longer the message, the higher or deeper the form.
An indicator, a group of five letters placed in the first position of the beginning of the message's cipher text, designated the key to be used on that particular day. The Japanese attempted to complicate further the solution of messages encoded in J-19 by establishing separate indicators (or "channels") for four distinct groupings of diplomatic stations: a general, worldwide audience, one for stations in Europe (which included diplomatic facilities in North Africa and the Middle East), another for those in both Americas (which included the United States), and one for stations in Asia. But American cryptologists had readily identified the indicator. This was the group that personnel at Bainbridge Island monitoring station recognized and therefore they were able to identify the J-19 cryptographic system used for the two messages of 19 November.
This allocation of audiences resulted in the situation whereby on any given day there were four additional sets of daily key in addition to the layout of the form that had to be recovered by the Americans. This process of recovering the key added to the difficulty required for the complete exploitation of messages encrypted in J-19 and its auxiliary transposition systems daily.[] Recovery of keys and the form often took well over a week. For example, the keys and form for messages of 18 November 1941 were not recovered until 3 December. By one estimate, at least ten to fifteen percent of J-19 key during the period leading up to Pearl Harbor was not recovered.[]
2.1.2. The Americans Solve the New Manual System
Shortly after the Japanese introduced the new manual code in mid-summer 1940, an Army code-breaking team headed by Frank Rowlett managed to isolate it in the intercepted diplomatic traffic. Frank Byron Rowlett was one of the first persons hired in 1930 by William Friedman for the newly hatched Signals Intelligence Service. Rowlett was born in southwest Virginia in 1908. After graduating from college, he took a job as a high school math and science teacher. In 1930 Friedman offered Rowlett a job as a "junior crypt-analyst," a position that was a mystery to him until he arrived at SIS. Rowlett demonstrated an ability to beat machine cryptography, solving the Japanese Red machine in 1935 and later supervised the team that broke Purple in September 1940. He also designed the major U.S. machine system known as Sigaba, a system that defied all Axis efforts to solve.
Figure 2.3. Frank B. Rowlett
There is some confusion, though, with the story of the solution to J-16 and that is in the vagueness of the chronology of the breakthrough. Rowlett, in his memoirs, relates that the new diplomatic system, which eventually would be labeled J-16, appeared about the time that the processing of Purple intercepts had gotten down to "a routine procedure." The first "Magic" translations were produced on 27 September 1940, but this "first" was achieved only because the two translations were of messages that used the same key. However, it would be about another three or four weeks before translations would be produced daily. The production had to await both the recovery of the Japanese method of key generation for Purple, as well as the construction of an analog device that performed all of the functions of the HINOKI machine.[] This would place the "routine" production of Magic sometime in the latter part of October.
However, available translations of Japanese diplomatic messages encrypted in MATSU indicate that the SIS team was exploiting the J-16K5 system well before the Purple breakthrough. While fragmentary, the dates of translations suggest the following chronology. As of late August 1940, a number of Japanese diplomatic facilities had begun to use J-16, notably the consulates in Seattle, Washington, and Honolulu, Territory of Hawaii. On 28 August the Japanese embassy in Geneva was informed that, as of the receipt of that day's message number #79, it was to begin using MAT[S]U. It also mentioned that eight other cities, including Washington, D.C., had already switched to MATSU.[] Within a week, a number of other stations were using MATSU or J-16, though some stations, such as Rome, were still using J-14 as late as 12 September.
Interestingly, the SIS was producing translations of messages in the new J-16 system as early as 7 September 1940, some three weeks before the first Purple machine translations were completed and several weeks before "Magic" translation production became a routine procedure. So, Rowlett's narrative, even as vague as it is, differs from the records. If, by late August, the Japanese were already using J-16 in eight cities, it could not have come "on-line" much earlier than that month. This means that Rowlett and his code-breakers did not have much time to analyze the new system and then solve it; in fact, it was only a few weeks.[] How did they solve it so quickly?
When the diplomatic traffic encrypted in the new system was studied, Rowlett noticed that the groups of characters in the new system differed in composition and frequency from those seen in the Japanese machine ciphers systems. He suspected a new manual system. Diagnostic tests that could derive clues to the system were applied to the groups in the intercepted text. The results suggested that a code was being transposed, but with a greater effectiveness than the predecessor systems like J-14.
Rowlett initially suspected that the Japanese were using a variation of a World War I German military field cryptographic systems known as "ADFGVX." The German Army implemented this field cipher as it prepared for the massive offensives of March 1918 that almost broke the Allied armies defending Paris. It effectively prevented the Allies from reading German radio traffic for several weeks until it was broken by the French in early June in time to stop the German onslaught. This was a system in which a plaintext message was encrypted with a digraph substitution cipher that used only the referenced six letters – hence the eponymous title. The message was inscribed into a form or tableau, but with nulls, that resembled something not unlike a checkerboard. The horizontal and vertical values, that is, the placement of the six letters, were scrambled every day. To further complicate the issue, the cipher text then was superencrypted using a transposition scheme. This last step had the effect of fracturing the original cipher text digraphs, thereby destroying the frequency of their incidence within a message – the best method for exploiting and solving the cipher, the frequency of certain digraphs, had been removed.[]
Rowlett reported that his team worked with an OP-20-G team on the effort against the new system, but that little progress had been achieved by either group. Then, about a month after the system had been in effect, probably late August 1940, Rowlett was invited by the Navy cryptanalysts in OP-20-G to visit them in their nearby offices. They revealed to him that naval intelligence (OP-16) had recently burglarized an unspecified Japanese diplomatic facility in the U.S. They had opened the code clerk's safe and photographed the codebooks, key, and other material. The haul included snapshots of the most recent J-16 codebooks, ten-day forms, and some of the key for the new auxiliary transposition system. From the pictures the Navy had taken, Rowlett now knew how the new J-16 cryptographic system functioned. He realized that the Japanese were encoding the plaintext and then transposing the resulting text. This technique broke up the code's digraphs and made solving it extremely difficult. Now, armed with copies of the documents photographed by the Navy, the solution and exploitation of the system, soon to have the J16K5 designator, would be much easier. This acquisition of the J-16 material explains how the first published translations of messages encrypted in J-16 were available in the second week of September.
The United States Navy's Office of Naval Intelligence (ONI) had compiled a long history of break-ins of diplomatic facilities and residences in various U.S. cities in order to obtain copies of cryptographic materials and other classified documents. Between the end of the First World War and 1941, Naval Intelligence carried out a number of these "black bag jobs." In the early 1920s, ONI had purloined the main Japanese naval code book to which the covername "Red" (not to be confused with the Red diplomatic cipher machine) was given. In 1935 the apartment of the Japanese naval attaché in Washington was burglarized, though nothing of value was found. Later, in May 1941, in one of the most brazen efforts, a team of navy and customs personnel boarded the Japanese merchant ship Nichi Shin Maru of the Pacific Whaling Steamship Company at Port Costa near San Francisco. They planted some drugs in the captain's cabin and in the ensuing confusion confiscated his copy of the current merchant (Maru) code and several other documents dealing with communications. When the Japanese consulate intervened and requested the documents' return, U.S. Customs replied they would give the documents back when the investigation was completed; presumably they meant the drug issue. The Japanese consulate informed Tokyo by cable of the situation. It also suggested to Tokyo that if any other Japanese vessels were boarded in the future, then the cryptographic material that was on hand should be destroyed lest it fall into American hands.[] The SIS, which had already broken that code, was angry that the Navy's stunt would alert the Japanese to American interest in their codes.[]
Despite the bounty, Rowlett was uneasy with the implications of the Navy's burglary and rightly so. While their photographic snatch had helped immeasurably in reducing the time needed to recover the new diplomatic code and its auxiliary transposition cipher system, Rowlett was worried about the long-term potential for compromise these actions posed for current and future Army cryptanalytic projects. If the Japanese ever suspected that their facilities had been entered and their cryptography had been compromised, they would change whatever systems were operational and, therefore, place him and his codebreakers back to the beginning. Moreover, if the Japanese became truly concerned over the scope of the compromise of their cryptographic systems, they might even go further and replace the Red and Purple cipher machines that had taken Rowlett and his codebreakers so long to exploit.[]
Rowlett went to the Army brass with his concerns about the Navy's break-ins. He saw Brigadier General Joseph Mauborgne, who was the Chief Signal Officer, head of the Army's Signal Corps, under which the SIS operated. General Mauborgne was an accomplished cryptanalyst and a long-time proponent of communications intelligence, going back to the early 1930s when he sat alone listening to his radio for foreign radio traffic while he was stationed in the army base in the Presidio in San Francisco, California. Mauborgne had been promoted to the position of the Chief Signal Officer in 1938 and had pressed for expansion of all parts of the army's COMINT program.
Mauborgne agreed with Rowlett's fears about the navy's second-story jobs. He wanted these clandestine forays into foreign diplomatic missions to end before a major compromise happened. He told Rowlett that he would "take it to the White House" if the navy refused to desist.[] However, Mauborgne, who was a technically gifted codebreaker in his own right, also had reservations regarding the Rowlett's ability to solve the new Japanese system without recourse to the Navy's "lifted material." In this, William Friedman, who headed up the SIS, seconded the general's doubts. Friedman believed that the new transposition cipher could not be broken by pure cryptanalysis.[] Still, the Army codebreakers went to the Navy Building next door to their offices in the Munitions Building on Constitution Avenue in Washington, and convinced the head of OP-20-G, Commander Laurance Safford, to get naval intelligence to agree to hold off for a while and to inform the Army about any future break-ins.
Mauborgne and Friedman's pessimism about the effectiveness of pure cryptanalysis against J-16 became a red flag waved in front of Rowlett. He was determined to prove the system could be recovered through pure cryptanalysis. He won a concession from both the Navy and Army crypto-logic staffs for time to allow him to attempt to recover any changes to the new system without any covert acquisitions. He banked upon the Japanese tendency to regularly replace current systems. He did not have to wait long. At the end of November, the Japanese replaced the auxiliary transposition system, K-5, with a new version, labeled K-6, as well as the basic code, which was known as HAGI (Shrub), which the Americans later labeled J-17. After two weeks, the Japanese slipped up. A message encrypted in K-6 was sent as a circular. A circular message is one that is sent to more than one station; in the case of this system, it probably meant that the message was sent to all of the stations in one of the four audiences. However, one station in this group received a version of the circular message with about fifty extra letters of text. This additional text allowed Rowlett to solve K-6. As it turned out, the K-6 transposition system merely reversed the indicators from the K-5 system and also inscribed text into the form beginning from the extreme right column instead of the left.[]
Over the next year, as each new variant to the original MATSU system was activated by Tokyo, the Army cryptanalysts were able to solve it. The code structure remained the same with plaintext values merely being reshuffled to new code groups. The auxiliary cipher systems were a variant of the preceding system. At the same time, independently of the Americans, the British and Dutch codebreakers in Singapore and Bandung were also exploiting the new manual system and its successors.
By August 1941 the Gaimusho cryptographers were ready to activate the latest variant. Rowlett and his team were tipped to the new system by a message from Tokyo to Washington on 22 June 1941. In the text Tokyo announced that the current code, SAKURA, known as J-18 to the Americans, would be replaced by the new version called FUJI.[] The first two messages in the new FUJI system were discovered to have been literally double superenciphered with the Purple machine. After the Purple cipher had been stripped away, the transposed code text was exposed. Rowlett analyzed the code groups after they had been transposed back to the original four-letter groups. Upon inspection of these groups, he recognized some curious combinations that led him to suspect that the groups were not from a table like the three earlier systems, but were derived from some other source. His suspicions centered on a letter count of the messages, which revealed that forty-eight percent of the text consisted of vowels. This did not square with the percentages from the previous system, J-18 or SAKURA.[]
Rowlett recognized that the coding system used in the FUJI appeared to be a derivative of a code known as "CA," which had been in use since 1936. The system had stopped using a cipher in 1937. Its use was often indicated by the appearance of the digraph "CA" in the first group. The system also had an auxiliary English speller table with twenty-six digraphs substituted for the letters of the English alphabet. This auxiliary system usually carried the indicator "AQ." This was another point of recognition that Rowlett may have locked onto during his analysis of the cipher. However, the first efforts to transpose the messages back to the original code groups failed.
Rowlett tried another approach, which was to recover the key and forms used in the messages. From these he derived the 1,600 digraphs of untransposed code. Then he developed an index for the recovery of the tetragraphs. By analyzing the new system in this manner, Rowlett looked first for the tetragraphs that would have indicated dates or foreign names. He found the tetragraphs in question; in fact, they were groups from the J-18 table which had been reversed. He broke the new system within a day.[]
However, solving FUJI, or J-19K10 as the code and its auxiliary transposition cipher were now titled by the American codebreakers, did not necessarily mean that exploitation of the traffic encoded in that system was easy. In fact, exploiting messages in J-19 remained a difficult proposition at best. Captain Safford estimated that the Americans failed to recover about ten to fifteen percent of J-19 key whereas only two to three percent of Purple key was not recovered.[] Actually, the key recovery rate was much lower. According to an OP-20-G cryptanalytic report from October 1941, as of the end of the previous month only twenty-one percent of the J-19K10 key had been recovered.[] The problem, of course, was that each day's key had to be recovered, while three times during a month the form or stencil was replaced. Estimates of time needed to exploit a message encoded in J-19 ranged from about a half day to as many as five, but individual messages could take anywhere from ten to fifteen days to decrypt.[] The irony was that, while the J-19 system was far easier to solve, it remained a considerably more difficult system to exploit daily. Purple took eighteen months to solve, but its exploitation was far easier – usually the decryption and translation of a Purple message were completed within one day of receipt of the original intercept, thanks in large part to the recovery by the navy cryptanalyst Frank Raven of the Purple daily key generating scheme.
The tradeoffs in the relative security of cryptographic systems sometimes belie their ultimate importance, as well as their vulnerability. The relative importance the Americans assigned to the exploitation of J-19 and Purple suggests that J-19 traffic was not considered as important to complete as was that for Purple. Precisely why is not clear. It is possible that the ease of solution for J-19 might have biased American codebreakers into believing that, in terms of importance, it was a secondary system relative to Purple. Also, that Purple machines generally were distributed to major capitals or cities might have led the Americans to consider the machine a far more valuable intelligence source. Whatever the reason or mix of reasons, Purple became and remained the priority Japanese diplomatic target for the army and navy codebreakers up to and beyond Pearl Harbor.
The resulting joint effort against Purple continued to consume the major portion of time and analytic resources available to both services. After Purple, the two services worked J-19, PA-K2, and the LA systems, followed by plaintext traffic and broadcasts. The efforts against messages in systems other than Purple suffered by comparison because of this prioritization. This deficit can be measured in terms of messages translated, the penultimate step in processing any intercept. For example, from 1 November to 7 December 1941, American cryptologists decrypted and translated about two-thirds of all intercepted Purple messages. During the same period, only sixteen percent of all intercepted messages encoded in J-19 were translated.[] Essentially, even if a message had been decrypted quickly, it could sit in an in-basket awaiting translation. There was, in the words of Captain Safford, "no urgency" attached to exploiting messages encrypted in J-19.[]
2.1.3. November 19: Japanese Message #2353 – The First Winds Instruction Message
The first message intercepted by the monitoring site at Bainbridge Island, Washington, Japanese message number 2353 Exhibit #3],[] was not completely processed until 26 November. It is not certain when it was decrypted, but there is some evidence that the British FECB in Singapore recovered the key to that day's J-19 cipher and relayed it to Washington via London on 24 November.[] The recovered key sequence Exhibit #5][] used to transcribe the coded text into the columns read: 3-17-12-4-5-18-2-10-19-7-11-9-14-1-6-16-13-15-8. The indicator BUTWJ designated the message for a global audience and that form (or stencil) #8 was to be used for the transposition cipher.
Since the message was intercepted on an odd-numbered day, it was OP-20-G's job to process it. The analyst's first step was to inscribe the code text into the correct Form, number 8. [Exhibit #56][]The next step was to correctly read out the code digraphs on a worksheet. But this required the daily key, which was not available until 24 November. On this sheet the analyst would write the Japanese kana plain text value under the code groups. Exhibit #6][] After this, a Navy linguist produced the kana text version of the message to translate. Exhibit #7][] A translation was finally published and released on 28 November. The translation carried two serial numbers representing the split/double duty by the two services. There was the Army SIS number, "25432," and the Navy JD-1 (Japanese Diplomatic Translation, 1941), "6875."[] Exhibit #8][]
The main points of message number 2353 were these:
The "execute" message phrase was to be sent in case diplomatic relations between Japan and one of the three named countries were "in danger."
There were three phrases, each unique and signifying the state of relations with one of the three countries:
Each phrase would be repeated as a special weather bulletin, twice in the middle and twice at the end of the daily Japanese language short wave voice news broadcast.
When the message was heard, each diplomatic facility was to destroy all codes and important papers.
Interestingly, the SIS revised this translation in September 1944. This was done at the request of William F. Friedman, who, at the time, was preparing to testify before the first round of hearings of the Clarke Investigation into the attack on Pearl Harbor. Three Army linguists worked on the new translation, including John Hurt, who had been hired by Friedman in 1930 as part of his original staff. The translation added some of the personal tone of the message missing in the original. The revised version differed little in the text except for one point. In terms of relations with Great Britain, it added that the situation also could include an occupation of Thailand, the invasion of the Netherlands East Indies, and the invasion of Malaya. Exhibit #9][] No one knows why these three additional scenarios were kept out of the original translation issued on 28 November 1941, especially since the reference to the Netherlands is obvious in the kana text.
2.1.4. November 19: Japanese Message #2354 – The Second Winds Instruction Message
The second Winds message, number 2354 Exhibit #4][], was decrypted and translated by 26 November, seven days after it had been intercepted. The message's encoded text was inscribed into Form #8. [Exhibit #4][] As with message #2353, the analyst recovered the true code text only after receiving the key from the British site in Singapore on 24 November. The analyst then produced the worksheet of the original code text with the plaintext kana values written underneath each digraph code group. [Exhibit #11][]The translation was then produced, reviewed, and issued on 26 November. Like its predecessor, the final translation of message #2354 carried two translation serial numbers representing the split/double duty performed by both services. There was the SIS number, "23592," and the OP-20-G serial, JD-1 "6850." [Exhibit #12][]
The lower serial numbers by both services indicates that message #2354 was completely processed and released before #2353. Why this occurred is not totally clear, though the fact that the second message was about forty groups shorter may have been a factor. Also, like the first message, in September 1944 a revised version was done at the request of William Friedman. The revised version, though, differed little in substance from the original. [Exhibit #13][]
The main points to message #2354 were these:
The warning was to be sent if relations were in danger of breaking down – "mortally strained" was how the 1944 version translated the expression.
Three single words were listed as the alert codewords. These words happened to be the same first word of the three code phrases contained in #2353:
HIGASHI (East) if it related to U.S.-Japan relations;
KITA (North) if it concerned Japan-USSR relations; and
NISHI (West) if it concerned Japan-Great Britain relations, which included the situation in Thailand, Malaya, and the Netherlands East Indies.
Each word would be repeated five times both at the beginning and end of the General Intelligence, or News Broadcast [IPPA JOHO]. In the 1944 version of the translation, the instructions stated that the words would be inserted in the General News Broadcast, which was a Japanese overseas news broadcast transmitted in Morse code. For an example of a transcription of this type of news broadcast. [Exhibit #14][]
The Americans were not the only ones to intercept and process these two messages. Cryptologists for Australia and Great Britain also collected, decrypted, and translated the same messages. They produced slightly different texts, as would be expected. For example, for message #2353, Eric Nave, a Royal Australian Navy linguist, translated the introductory paragraph this way:
Owing to the pressure of the international situation, we must be faced with a generally bad situation. In that event, the communication between Japan and the countries opposing her would be severed immediately. Therefore, should we be on the verge of an international crisis we will broadcast twice....[]
On 28 November the Commander-in-Chief of the U.S. Asiatic Fleet (CINCAF), Admiral Thomas C. Hart, who later headed an inquiry into Pearl Harbor from February to June 1944, relayed the news to Washington and Honolulu that the British at the FECB in Singapore had intercepted the same two Winds messages. The message added that the British and the U.S. Navy monitoring station at Cavite, Philippines (known as "Cast"), would be listening for the two broadcasts as outlined in the messages. The message noted that the intercepted messages contained warnings that were to be broadcast if relations between Japan and any of the three other countries were "on the verge of being severed." [Exhibit #15][]
The Dutch intercept station in the Netherlands East Indies, Kamer 14, also intercepted and decoded the same two messages as the British and the Americans. The first, Japanese serial #2353, was transmitted from Tokyo to diplomatic stations in the Far East. The second, serial #2354, was relayed from the Japanese embassy in Bangkok to their consulate in Batavia on the island of Java. The Dutch authorities passed along the contents of the Japanese messages to both the American military and naval attachés in Batavia. Both attachés cabled the War and Navy Departments of the Dutch intercept. The State Department also was alerted to the messages by its representative in Batavia, Consul General Walter Foote. [Exhibit #55][] Foote reported the two Dutch translations, though he stated that the coded phrases meant "war" with either of the three named nations. Yet, in the same message to Washington, Foote added that the second message from Bangkok to Batavia carried the expression "threat of crises." Foote, however, was skeptical of the importance of the information from the Dutch. He noted in his report that since 1936 such warnings of impending Japanese hostilities in the region "had been common."[]
It is worth emphasizing that the Japanese Foreign Ministry established two distinct, though related, ways of notifying its diplomats of a change in relations that warranted the destruction of vital papers and cryptography. The Gaimusho intended to set up a warning mechanism that accomplished three things. First, it would be effective even if traditional lines of communications were cut off. Tokyo's diplomats could listen over any shortwave radio for the broadcasts. Secondly, it was a mechanism that was unique for its intended audience. The scenarios spelled out in the Winds instruction messages, in which the open code (or codeword) phrases or words would be passed, were distinct from any situation in which a phrase or word about the weather could be misconstrued, i.e., a regular weather report or broadcast.
Finally, that anyone else might hear the open code phrase or word was not important. The meanings of the Winds codewords or phrases were innocuous to anyone else who might be monitoring the overseas broadcast out of Tokyo. The security of the mechanism was that the knowledge of the true meaning of the Winds code was restricted to the Japanese Foreign Ministry and its diplomats. The sense of security was heightened by the fact that the Japanese were certain that FUJI (J-19) could not be exploited.
However, in their certainty, the Japanese diplomats were gravely mistaken. Although the code phrases and words were difficult to exploit quickly, the Americans (and British and Dutch) within a week knew in detail the instructions contained in the Winds phrases and words. With this information in their hands, the Americans now had a way of measuring any change in the relations between Japan and the United States. A Japanese news broadcast that contained the Winds code phrases and words signified an increase in "danger" in relations between the two nations. The Americans knew that the next step was for Tokyo's diplomats to destroy all of the classified material and equipment held in its facilities. What could be a clearer warning than that?
What remained to be done was for the SIS and OP-20-G to task their respective monitoring sites to listen for the Winds message(s) and then await their transmission. Yet what was apparently simple would, over the next ten days, become complicated and confusing.
2.1.5. Notes
[] Frank B. Rowlett, The Story of Magic (Laguna Hills, CA: Aegean Park Press, 1998), 174; Multinational Diplomatic translation (MNDT) #9719, Tokyo to Washington, 13 August 1940. RG 457, Entry 9032, Box 291.
[] MNDT # 6938, Bogota to Tokyo, 24 April 1940 and #6906, New Orleans to Tokyo, 24 April 1940. RG 457, Entry 9032, Box 288.
[] Department of Defense, The "Magic" Background of Pearl Harbor (Hereafter "Magic" Background), Vol. I (February 14, 1941 – May 12, 1941) (Washington, D.C.: USGPO, 1977), A-101 to 104
[] It is not clear how Tokyo first informed its diplomatic facilities of the change of the codes to J-16 (MATSU). Frank Rowlett says that a circular message sometime in June encoded in an old system tipped off the SIS to the change. But a review of SIS translations from the period (Entry 9032, boxes 287–289) does not uncover such a message. There are some scattered messages from Tokyo instructing stations to begin using the MATSU system, but these are dated from late August and early September 1940.
[] "Japanese Diplomatic Network and Crypto Systems, Pre-During (sic) the War." RG 457, Entry 9032, Box 992, Folder 3015.
[] "Auxiliary Diplomatic Systems." R.I.P.-37B, 15 September 1940, 9–1. RG 457 Entry 9032, Box 1137, Folder 3671
[] "Change No. 4 to R.I.P. 37B, "K10 Transposition System." 1 April 1944. RG 457, Entry 9032, Box 1137, Folder 3672, 7–119
[] Ibid., 7-58 to 7-59
[] "J-19 System Description (Draft)." RG 457, Entry 9032, Box 992, Folder 3015, "Jap Diplomatic Network Cryptosystems; Pre and During WWII." 2–3
[] Wohlstetter, 175
[] For a description of this process, see Alvarez, 80–82 and Stephen Budiansky, Battle of Wits. (New York: The Free Press, 2000), 164–168. Almost all narratives of this event draw heavily from William Friedman's "Preliminary Historical Report on the Solution of the B Machine," 14 October 1940, "History of Japanese Cipher machines," RG 457, Entry 9032, Box 808
[] MND Translation #10437, Tokyo to Geneva, 28 August 1940. RG 457, Entry 9032, Box 291
[] Rowlett, 175.
[] For a more detailed narrative about the solution and exploitation of the ADFGVX cipher, see Kahn, 341–345; Also War Department, Office of the Chief Signal Officer, Washington, General Solution for the ADFGVX Cipher System: Technical Paper. (Washington, U.S. Government Printing Office, 1934). For information about the diagnostic analysis and solution of transposition ciphers, see Lambros D. Callimahos and William F. Friedman, Military Cryptanlaytics, Part II (Fort George G. Meade, MD: National Security Agency, Office of Training Services, 1959), 415–434.
[] MND Translation #18037, San Francisco to Tokyo, 28 May 1941. RG 457, Entry 9032, Box 738, Folder 1812, "Jap Suspicions that U.S. [and] Allies Reading Codes, (1940–42)"
[] For more on the Navy's program of stealing or copying codes, see Stephen Budiansky, Battle of Wits, 84; Rear Admiral Edwin T. Layton with Captain Roger Pinneau and John Costello, And I Was There: Pearl Harbor and Midway – Breaking the Secrets (New York: William Morrow and Company Inc., 1985), 109–110; Frederick Parker, Pearl Harbor Revisited: The United States Navy Communications Intelligence, 1924–1944 (Fort George G. Meade, MD: NSA, Center for Cryptologic History, 1994), 4–5; and Ellis Zacharias, Secret Missions: The Story of an Intelligence Officer (New York: G.P. Putnam's Sons, 1946), 178–82
[] Rowlett, 180–1
[] Ibid., 183
[] Ibid., 182–3
[] R.I.P. 37B, 7–33
[] MND Translation, Tokyo to Washington, Circular #1295, 22 June 1941. RG 457, Entry 9032, Box 992, Folder 3015, "Jap Diplomatic Network Cryptosystems; Pre- and During WWII."
[] "J-19," Ibid. Frank Rowlett's Notebook, also in Box 992.
[] "J-19," Ibid. For more on the CA system, see R.I.P 37A, "General Characteristics of Japanese Diplomatic Systems," Part I, Section 5–25. RG 457, Entry 9032, Box 1137, Folder 3670.
[] PHH Part 36:314; also in Wohlstetter, 175
[] "Monthly Progress Report for September 1941," RG 38, Entry 1040, Box 115, Folder 5750/198, "GYP-1 History"
[] Ibid.; Kahn, Codebreakers, 16–18
[] PHH, Part 37:1081–83
[] "Friedman notes of meeting with Captain Safford," page 11. RG 457, Entry 9032, Box 1360, Folder 4217. "Pearl Harbor Investigation and Miscellaneous Material."
[] CCH Series XII.S, Box 22 and "Jap Msgs, oct-Dec 1941," RG 38, CNSG Library, Box 156, Pages 3803–4.
[] CCH Series XII. S, Box 22. SIS #23542, 28 November 1941. William Friedman Marginalia: "Key furnished by British (Navy says Singapore to London to Washington 24 Nov[ember] [19]41). F[riedman]
[] R.I.P. 37b, 7–83
[] NARA, RG 80, PHLO, Entry 167A, "Office Reference ("Subject") Files, 1932–1946." Winds Code, Station "W" to Witnesses. Folder: Winds Code - Misc Material.
[] Center for Cryptologic History, Series XII.S, Box 22 and NARA, RG 80, PHLO, Entry 167A, "Office Reference ("Subject") Files, 1932–1946." Winds Code, Station "W" to Witnesses. Folder: Winds Code - Misc Material. A copy of the same worksheet also is available at NARA, RG 457, Entry 9032, Box 1360, Folder 4217, "Pearl Harbor Investigation and Miscellaneous Material."
[] GSB 180, 6 November 1941[5]. RG 38. CNSG Library, Box 166, Folder 5830/69, "Winds Msgs."
[] The Army SIS used a simple one-up serialization scheme for its diplomatic translations beginning in 1939. The SIS also included a prefix to this numbering system. During the war, the prefix changed three times – "SSA" to "SSD" to "H" – but the numbering system continued. The Navy's OP-20-G used a more complicated system for diplomatic translations. The serial consisted of a trigraph indicating the country and element, in this case "JD" for Japanese diplomatic messages. The third element in the trigraph was a single digit representing the last digit in the year that the message was translated. So "JD-1" indicated that the translation was of a Japanese diplomatic message completed in 1941. This digit changed with each year, so "JD-2" indicated a Japanese diplomatic translation completed in 1942 and so on through 1945. The serial number after the trigraph was the serial of that translation. The Navy would reset the serial number to "0001" on 1 January of the new year. Translations of Japanese naval messages used the same method – "JN-2" indicated a translation completed in 1942.
After the war, OP-20-G decrypted and translated Japanese naval messages it had intercepted prior to the start of the war, emphasizing those relevant to Pearl Harbor. These messages all bear serials with the tri-graphs of "JN-5" or "JN-6" meaning they were completed in 1945 or 1946.
[] CCH Series XII.S, Box 22
[] Ibid.
[] Ibid., and NARA, RG 80, PHLO, Entry 167A, "Office Reference ("Subject") Files, 1932–1946." Winds Code, Station "W" to Witnesses. Folder: Winds Code - Misc Material.
[] CCH Series XII.S, Box 22 and "Jap Msgs, Oct-Dec 1941." RG 38 CNSG Library, Box 156, page 3798.
[] Ibid.
[] Ibid. A copy of this translation, without Friedman's inscribed comments, can be found in the Multi-National Diplomatic Translation collection, SIS # 23592, Tokyo to Washington, 19 November 1941. RG 457, Entry 9032, Box 301.
[] Ibid.
[] Japanese News Broadcast by station "JVJ," 8 December 1941. NARA, RG 38, CNSG Library, Box 167, Folder 5830/69 (3 of 3), "Pearl Harbor Investigations: Winds Msgs."
[] James Rusbridger and Eric Nave, Betrayal at Pearl Harbor: How Churchill Lured Roosevelt into Wordl War II (New York: Summit Books, 1991), 135–6. This example of differing translations points to an issue raised in a book by Keiichiro Komatsu, The Origins of the Pacific War and the Importance of 'Magic' (New York: St. Martin's Press, 1999). Professor Komatsu claims that the translations produced by the American cryptologists often exaggerated, mistranslated, or misconstrued the original Japanese text, thereby misleading American leaders as to the true intent of Tokyo, which, as he claims, was for a negotiated way out of the confrontation.
Yet, the Roosevelt administration in Washington had problems assessing Japan's ultimate intent in the Far East. There were many causes, including the technical difficulties in rendering subtle Japanese diplomatic text into an English version that was faithful to the original and conveyed that same meaning to the Americans. But the effect of Japanese actions were more important. The civilian government in Tokyo failed adequately to control the Imperial Japanese Army's actions in China. As a result, Tokyo's policy often appeared to the Roosevelt administration as erratic, deceptive, or at cross-purposes. Also, it must be understood that American policymakers were reacting to nearly ten years of overt Japanese aggression in Manchuria and China. Likewise, the two-stage takeover of French Indochina by Japan (1940–41), and the American economic sanction in response, which came to be the probable major immediate casus belli for the Pacific War, could only be interpreted within the same context of previous Japanese aggression in China. The cause of the Pacific War was far more than a matter of how intercepted Japanese diplomatic dispatches were being translated. Rather, it was the aggressive Japanese policy of aggrandizement throughout East Asia that lay at the heart of the dispute between Washington and Tokyo and ultimately led to hostilities.
[] CINCAF Intelligence Report, 281430, 28 November 1941. PHH, Part 17:2660
[] ALUSNA, Batavia CR0222, 031030 December 1941. NARA, Washington D.C. RG 128.3, Exhibit 142, Box 334; also in RG 38, CNSG Library, Box 166, Folder 5830/69.
[] Batavia to Secretary of State, Washington, DC, 4 December 1941. NARA, RG 38, CNSG Library, Box 166, Folder 5830/69. Interestingly, in an article in 1986, James Rusbridger claimed that the British and Dutch had both intercepted the Winds Execute message on 4 December. However, Rusbridger then stated that the message, which was sent in two parts, was transmitted to Washington by the British monitoring station in Singapore and that the Dutch had passed their intercept to the U.S. consul-general, who, in turn, relayed the intercept to the War Department. Aside from the misconceptions about the flow of information, that is, Singapore would have sent the information to London and/or the US Navy site at Cavite, Philippines, and not directly to Washington, Rusbridger has confused the two messages intercepted on 19 November with the alleged Execute message. In fact, in his description of the exchanges between the Americans, British, and Dutch actually is that of the exchanges of the two Winds instruction messages. See "Brit Researcher Confirms Briggs' Winds Execute Intercept." NCVA Cryptolog, (Volume 8, Number 1, Fall 1986), 1–2, 21.