Breaking Historical Ciphers: An Emerging Team

Breaking historical ciphers in an interesting field; it combines two rather different disciplines, cryptanalysis and history. Where history has touched on cryptology in the past has been where codebreaking has had some historical importance, such as the Allies breaking of the German and Japanese codes and ciphers during World War II. Cryptological history is another historical discipline that unfortunately lives largely in the shadows of historical research. Most historians probably know little about it and for some it probably does not exist.

Programs for breaking historical ciphertexts is therefore a unique chance for those of us who are interested in the cryptanalysis of classical and historical codes and ciphers and the history of cryptology. The field is very promising for many reasons. Firstly, for the cryptanalysts, both amateur and more seasoned actors, there is a chance to attack real systems used in the real world. The effort will give the cryptanalyst new and detailed insight into classical cipher systems and methods, which might reveal new information about their strengths and weaknesses. Furthermore, attacking hard cryptanalytical problems will also bring the reward of developing new and refined methods of solution. The final reward it to discover a previously unknown or undisclosed plaintext that might be of historical significance. Secondly, if the recovered plaintext is of historical or public interest then there will  be the added reward of media interests and especially academic interests among historians and other working in related fields. This media interest might open the eyes of people who firstly will discover the work done by the small community of historical cryptanalysts, but secondly and more importantly they may wish to take part in the work on breaking historical ciphers.

To prevent historical cryptanalysis to end up in the box marked “Dying or Extinct Species” there is an urgent need for help outside the cryptological field. We need historians and others, such as the many ‘archive rats,’ to tell us about interesting code- and ciphertexts that we can attack. To further this symbiosis there is a need for all of us to make an effort to inform at all levels about this interesting work. Therefore I ask you to please go out and become a missionary for historical codebreaking. Stand on the roof tops and shout: “We need historical ciphertexts. Now!”

Last year there was an attempt that originated largely in the academic fields of philology and history to get funding from the European Union for a program for historical cryptanalysis. An application for funding from COST, European Cooperation in Science and Technology, was made for a project called HICRYPT,   “Historical Cryptology – Unlocking Europe’s Encrypted Heritage.” Unfortunately the project did not get funding last year, but perhaps there will be other opportunities. The world’s economic situation is of course not favourable at the moment for projects that have rather weak foundations both in the academic world and elsewhere. It is therefore very important that when you get significant results from your historical codebreaking you inform the public and other interested players.

HICRYPT had as aim to decipher rather old encrypted historical texts such as the 250-year old text deciphered by Christiane Schaefer, Wolfgang Hock and Kevin Knight and described in Wired Magazine in November 2012, “They Cracked this 250-Year-Old Code and Found a Secret Society Inside.” This story made headlines around the world and it is the kind of break that makes waves well outside the academic communities and the small world of historical codebreakers.

Now you might say, didn’t he get his title wrong. Should it not be an “An Emerging Field” instead of “An Emerging Team.” Well, hopefully it will develop into an emerging field but my intention is to pay tribute to a small team of people who during the last few years have made significant progress in breaking historical cipher systems. They are George Lasry, Nils Kopal and professor Arno Wacker. Professor Wacker is the head of the research group “Applied Information Security” at the University of Kassel, where Nils Kopal is a Ph.D student and where also George Lasry now works on his Ph.D.

The team has already published three excellent articles in the journal Cryptologia, “Solving the Double Transposition Challenge with a Divide-and-Conquer Approach,” “Automated Known-Plaintext Cryptanalysis of Short Hagelin M-209 Messages,” “Ciphertext-only cryptanalysis of Hagelin M-209 pins and lugs,” and other publications are being prepared. However, before you dive in to study these interesting articles I would advice you to set aside one hour and listen to a talk George Lasry presented to the students and staff at the University of Kassel in October 2015, “Cracking Unsolved Historical Ciphers.” His talk is worth listening to and you probably will, like me, be inspired by his enthusiasm and great love for cryptology  and cryptanalysis. The rush of adrenaline you experience when you break one of these ciphers and see the plaintext starting to emerge is, as he describes it, a unique experience.

Having payed tribute to this outstanding team I should nevertheless add that there are others out there who are just as dedicated and who also have my great admiration. Some of them like to keep out of the limelight but that does not mean that they are lesser cryptanalysts, a few of them are simply amazing. I am honoured to count them among my friends. And I must not forget Klaus Schmeh for his tireless work of discovering new crypto challenges and new historical texts with cryptographic puzzles. If there is one person who keeps us supplied with cryptograms it is Klaus Schmeh.

And I should like to thanks my friend Christos who made me discover the talk of George Lasry, “Presentation on Solution of Historical Ciphers.” Thank you Chris.

Captain Walter J. Fried’s Fish Reports

Captain Walter J. Fried (1904–2003) was the US Army’s liaison officer at Bletchley Park (BP) in the period March to November 1944. Walter J. Fried was born in Lawrence on Long Island, New York in 1904. He graduated with magna cum laude from Harward in 1924 and received his law degree from Columbia Law School in 1928. He started his career as a lawyer by joining his father’s law firm in 1929. For more details about this life and career see his New York Times obituary:
Walter J. Fried, 99, Lawyer Involved in Co-op Conversions

In March 1944 he took over as US Army liaison officer at BP after his predecessor Lieutenant John Norman Seaman (1914–2002), who was liaison officer at BP from August 1943 to March 1944. Lieutenant Seaman, who was also a law graduate, returned to BP as liaison officer in May 1945 where he participated in the TICOM operations there and in Germany. He left the Army as Lieutenant Colonel in 1946 and continued his career as a lawyer.

Walter J. Fried’s first report, F-1 (IL 3331/A), is dated 11 March 1944, subject: Dutch Hagelin (NEA). He had then been working together with John Seaman to acquaint himself with his BP liaison duties. His last report is F-123 (IR4070) dated 29 November 1944, subject: Miscellaneous Items. He says he expects this to be his last formal report, which indeed it was. His sucessor was Albert W. Small (1910–1966, who was liaison officer from November 1944 to May 1945. Albert Small arrived already in October 1944 and worked with Walter Fried in the overlap period. The Fried reports F-114, dated 13 November, and F-115, dated 17 November, are both written by Albert Small while Fried was in Paris. On 11 November 1944 Fried was ordered to go to Paris for a period not to exceed two weeks.

Captain Walter J. Fried was a prolific report writer turning out reports of an extraordinary quality. For the historian of cryptologic history they are extremely valuable due to their very detailed overview of cryptanalytical operations, discoveries and problems taking place at BP during the period March to November 1944. The subjects treated and the technical and historical details that these reports contain set them out as some of the best cryptanalytical documents from this period.

Interesting is also the correspondance between Walter J. Fried and William F. Friedman showing the close friendship between these to US Army cryptologists. The correspondance is available here:

Fried–Friedman Correspondance

Hopefully I have got your attention and wetted your interest. If you want to study Captain Fried’s Fish Reports you should visit the page I have just created on CryptoCellar with his Fish Notes:

Captain Walter J. Fried’s Fish Reports

For the moment only three of the reports are there, F-46, F-68 and F-116, but I plan to publish others as soon as I get the time. Please visit the page again soon.

Update: As of 16 February 2016 there are now 22 Fish Notes online. Still missing is F-91 and the appendix of F-71. These will be added later.

CryptoCellar has again risen from the ashes

Like the mythological bird Phoenix my Web pages CryptoCellar at cryptocellar.org has risen from the ashes. Unfortunately it has taken longer than I anticipated. It is still the old, but nevertheless revised, pages that are back. It will probably take a lot more time before I have a new design ready. The ideas are there but the time and energy is lacking. At least I am now feeling that I am back on track and if I am not get too much sidetracked there are still hope that I will be able to present you with some new material from time to time. At least I should try to be a little more active on these pages, submitting the occasional news and ideas from the CryptoCellar world.

In the meantime please have a look at the old pages and see if there is still something there that you find interesting. Most of the stale links have been repaired and a few new ones have been added. I am sure there are still some sore points here and there, but I will try to deal with those in the days and weeks to come.

Back to the Future or Where is Cryptocellar.org?

I knew it would happen one day but I always pushed the thought away, always propelled it into the Future. What I don't have to do today I can do tomorrow or perhaps next week, next month, next year, ... The Future is endless.

Since I left CERN I knew that my Web pages at cryptocellar.org, which have been hosted free on a number of CERN Web servers since I started it in 1995, were hanging in a thin thread. When I retired in April 2012 I lost access to all CERN computers and servers including the Web server hosting my CryptoCellar Web pages. This meant I could no longer maintain the site, add new material or correct stale links. The pages lived their own life outside my control, however I knew the pages were still serving a purpose. They were still being visited by people and on 10 September 2015 the Web counter showed 779514 visits.

On 23 October 2015 I received an e-mail from a friend, he wrote:

I just went to look at something on your site, but it appears to be down. Is this a temporary outage?  Or did CERN kick you off their servers?

Sadly it is the last of his questions that can be answered with a Yes. CERN has now got their own Internet domain, .cern, and in the process of moving all their Web servers to the new domain they are also changing their Web hosting policies. From now on only persons actively engaged in CERN activities can host Web pages and the content of these pages will from now on also have a much narrower profile than previously. Cryptology is no longer seen as a subject connected with CERN and its core activities.

So my Web pages has disappeared down the drain and into the dark Internet void of deleted Web pages and closed Web servers. It means I must now start in ernest the work of finding a new hosting service for my CryptoCellar Web site and perhaps also let the pages get a new look and feel.

However there is a immediate saviour, The Wayback Machine, which has archived a complete copy of my Web pages. While you wait for me to get my new CryptoCellar up and running you can access the archived copy here: CryptoCellar Archive

Urgent Warning to Enigma Owners

Enigma Corrosion Alert

Tom Perera, the owner of the Enigma Museum — a company devoted to locating, restoring, preserving, and trading German Enigma machines as well as antique cipher, telegraph, scientific, and communications devices — has just issued an urgent warning about a serious case of corrosion in some Enigma machine. The source of the corrosion seems to be coming from one type of the plastic light filter that these particular machines are fitted with. The light filter is usually fitted in the lid of the machines and it is strongly suspected that these filters are generating Acetic acid vapours that will build  up within the case when the lid is closed and then attack all close by nickel and copper parts.

I hope this photo will convince you of the seriousness of this corrosion problem and that it will make you visit Tom Perera's Enigma Corrosion Web page for more detailed information and how you can prevent this happening to your own Enigma machine(s). 

Crypto Cellar Tales is NOT Dead

Many of you probably are wondering what has happened to Crypto Cellar Tales. A Blog that does not publish anything must be dead. This is usually the case but it is not the exactly true for Crypto Cellar Tales. I warned you in my first posting that I probably would post in an erratic fashion or to say the truth: only post when I really had something to tell. Cracking jokes I usually do on Twitter. That it would take me more than a year to make my next posting was neither intended nor wanted, but when you lead a busy life with many interests and many commitments and on top of that you are someone who is rather slow, the outcome is given.

The important thing is that I am now back and I hope to be able to post somewhat more regularly. I know now that I have things to say that cannot be said easily on Twitter so in the next months or so I hope there will be more tales to tell. 

PRISM, Metadata and Unwarranted Spying

With my strong interest in cryptography and cryptanalysis I also follow closely what is happening in the signal intelligence fields and I am especially interested in what the big players, NSA and GCHQ, are doing. I have therefore found this week's revelations about NSA's PRISM program, their collection of phone companies so-called metadata and the Offensive Cyber Effects Operations (OCEO) to be both interesting and deeply disturbing. 

None of these programs really surprises me. I have seen it coming for a long time and there were sufficient indications over the years to fully expect such programs to be already running. However, that surprised me more was the scope of these programs. I have always expected that they would go after the bad guys and those under serious and warranted suspicion of being terrorists or national security risks. Now we learn that we are all suspects and that you are regarded as guilty until proven innocent. The problem is that as you do not know what they do with the data related to your modern, digital life and you have no idea of what tag they apply to you, terrorist, suspected terrorist, terrorist connections etc., you are not able to put up any defence to prove your innocence. You have just become a number, a pointer, a link in their anti-terrorist network and if your number, pointer value or link happen to reach any of the magic thresholds they have in their algorithms you will find that your life suddenly changes. You will suddenly receive extra attention in your daily life, traffic police will flag you down to check your papers and your car, when you fly it is always you that are pulled aside for extra screening etc. Some might fancy such attention but I do not. Because it might not just stop there, if you are really unlucky you might no longer be able to buy any airline tickets, your number has been elevated to inclusion in the very exclusive set, the no-fly list.

You will say it is very unlikely anything like this will happen to you. If you do nothing illegal you will not be targeted. The problem with this argument is that I don't know if I do something "illegal" when I am living my daily digital life, blogging like now, twittering like I did a moment ago and added a few new connections to my Facebook page yesterday evening. Today we make connections and new acquaintances by the dozen every week, how do you know that not one of them is a terrorist or has terrorist connections. In the last 20 years I have corresponded with a lot of people I don't know. I have been helping students in Pakistan, India, Irak and Egypt with information about cryptography for their school work. Were any of them connected with terrorists or criminals in any way? I don't know. To me they were just young students that I felt I should help with the same open door policy that I have given those coming from USA, Germany and England. Perhaps they were bad girls too, (yes surprisingly enough girls are also interested in cryptology). I don't know. But somebody knows and this somebody is sitting in front of a computer screen in an operation room at NSA. There they have just elevated my number, my name, onto one of the watch lists. The next wrong move and I creep a little higher until I hit one of the dreaded thresholds. Then the bell rings, the e-mails to the national security services are sent and the circus starts. And when it is started it might never stop. We all know how difficult it is to get removed from an electronic database, it is a never ending uphill struggle; and when those databases are secret ...

Now Obama and Mr. Clapper, his Director of National Intelligence, say it is only metadata we don't listen to your calls. Well, I think that is the real problem. If they had listened to my many calls to the "supposed terrorist" who has just sold me a used car, they would quickly understand that I am arguing about the price, especially after the transmission broke the week after I bought it. Instead my many and increasingly angry calls to the "terrorist" increases the connection counter on every call and the longer the calls are, the higher the weight they get. At then at the end of the week I am on the suspected terrorist watch list. You don't believe me; well try buying a used car from a terrorist and you will see the fun you will get. To get a better feeling for metadata have a look at this.

Am I paranoid? No, I don't think so, just worried. I don't like this way of collecting intelligence. Just because it is possible to do so now it does not mean it is should be done. 50 years ago all this would have been unthinkable. The only way the intelligence services then could have collected this kind of information would have been for every telephone switchboard to log all possible connections, all post offices taking notes of the addresses of all letters, parcels and postcards passing through their doors. The local police would have to visit everybody's home to search through their belongings, listing all the books on their shelves, all their records they own and carefully go through their diaries, agendas and photo albums. I doubt very much the population 50 years ago would have accepted this. It would have been a revolt and that is why there now has to be a revolt against this kind of unwarranted dragnet intelligence that sweeps up all and everybody in their nets. President Obama claims all this is legal because there is something called the Patriot Act. Perhaps it is time to look closer at this Patriot Act and see if it is as constitutional as it claims to be. To me it seem like the Patriot Act surreptitiously has rewritten the United States' Constitution.

I am not against intelligence collection, even when I am the target, as long as there is a very good and lawful reason for the collection. And when I say lawful I don't mean any quickly cooked up anti-terrorist laws, but laws that respect international law and Human Rights. Anything else I would regard as unwarranted and unlawful. President Obama, I think you will have your hands full the next few months as this simply will not go away. Neither should it. It is too important for that. At the moment we are on a very dangerous and slippery slope. At then end is the horrors of 1984 and Dystopia. Some will say this is the just price to pay for eradicating terrorism. I think not, because terrorism can not be eradicated, only controlled and I am convinced Dystopia would just create more terrorism and this time of the homegrown sort. Take your pick, it will be hell either way.

So please high priests in the governments around the world don't push your intelligence agencies into this kind of quagmire. I have met many intelligence officials over the years and several I consider as very good friends. There are very few I have ever met who I think would enjoying targeting innocent people in the way it is done today. I think many intelligence officials are living increasingly difficult lives with their conscience bothering them for the rest of their lives. It was easy as long as the targets were Nazi Germany, the Soviet Union and North Vietnam. When the target is the family down the street it quite another game. 

We need good and just intelligence to protect our democracies but we should be very careful not to undermine our democracies with unwarranted spying just because the tools are there. Today a major part of our lives are in the digital world. Any responsible government should do its utmost to protect and guard our digital souls and not trample on  them. We expect nothing else. Good and strong encryption should be mandatory for everybody in the same way as vaccination is a part of our health protection programs.

President Obama you have some serious work to do if you want to remain credible.

UPDATE.

As I expected this has now developed into a serious debate both in the USA and abroad. I am extremely happy about this turn of the events because I think it is very important that we are aware of what is happening in the surveillance fields and the directions it now takes. Previously the targets were governments and states. We, the citizens of these states, were nothing more than spectators to the spying games. When revelations would appear in the media from time to time it would only bring us some degree of awe or excitement in our daily life. Unfortunately, the targets are no longer only government and states, now also YOU are in the cross-hairs. The surveillance net has tightened its masks to catch also the single individuals all over the world.

The amount of information that is in the press at the moment is staggering and it is not at all easy to find your way between the truths, half-truths and the clearly wrong information. However, I have just found one article that I think is very well balanced and which I think is getting closer to the real situation. Therefore have a look at Marc Ambinder's article "NSA: Sucks in data from 50 companies."

I will quote one paragraph that I found illustrates the problem for us non-US citizens:

And the government insists that the rules allowing the NSA or the FBI to analyze anything relating to U.S. persons or corporations are strict, bright-line, and are regularly scrutinized to ensure that innocents don't get caught up in the mix. The specifics, however, remain classified, as do the oversight mechanisms in place.

At least the way it is presented here seems to suggest that non US persons or corporations will a priori not have the same protection with regard to innocence. I think we are regarded as fair game, which means that we can freely be scrutinised to whatever extent necessary to see if we are a security threat or not. We are all guilty until proven innocent. With a few strokes of the pen we have been stripped of all our privacy and rights.

Here is a great timeline: Timeline of NSA Domestic Spying