IBM's SoftLayer cloud infrastructure service blocks Cuba -- why now?

Cachivache Media recently reported that the Bitly URL-trimming service had stopped working in Cuba. Cubans had been using the service for several years, so this resulted in many broken links.

Cachivache did not know what had happened, but published a traceroute that timed out at an Akamai router. I contacted Akamai, and they said they could not say anything -- they would only talk with their customers -- Bitly in this case.

This traceroute from Cuba to Bitly times out in the Akamai network.

So I contacted Bitly and had an email exchange with one of their support people. (The press and operations departments failed to answer my emails and I could not find a phone number to call). This is a transcript of my email conversation with their support representative:

Larry: My colleagues in Cuba are unable to reach their bit.ly account. They say it failed some time ago, worked yesterday and is now broken again. I attach a traceroute.

Support: Unfortunately, Bitly links do not function correctly in Cuba. This is not an issue on our end – I believe that Cuba and Iran are both unable to access Bitly links, due to government regulations.

I wish I had more info! Let me know if you need help with anything else.

Larry: Cubans have been using Bitly for years and they are no longer on the list of state sponsors of terrorism -- it just recently became unreachable. It was back up for a day earlier in the week then went down again. There is some sort of intermittent failure.

Could you follow up with Akamai on this? Or, if it is a change in your company policy, could someone confirm that?

Support: Thanks for getting back to me. Unfortunately there is not much I can do here, we’ve had reported problems with our links in Cuba, and are working diligently to rectify the issue.

Larry: I am confused -- are you now saying that it is a technical issue rather than policy? If so, by when do you expect to rectify it? The traceroute times out at an Akamai router -- have you filed a help ticket with them?

Support: I wouldn’t necessarily say this is an issue on our end. We know that our links don’t always work in Cuba – we’re not in touch with the Cuban government about this however.

I really wish I had a better answer for you, but I don’t unfortunately! I hope you still find value in our free tool.

Larry: Are you doing it in compliance with a request of the US government? Is Akamai?

Support: As I mentioned, we’re aware of this issue, our engineers are aware and are working to solve the problem.

I can’t provide any more additional info at this time, I apologize for the inconvenience.

Well, that was inconsistent, but I guess a tech support person does not have authority to answer such questions.

Next, I heard from a friend in Cuba who told me it was not only Bitly -- other sites that used Bitly to trim their URLs were also blocked. Confused, I asked a colleague, Doug Madory, who monitors the Internet at Dyn Research, what he thought was going on. It turned out Doug had also been looking into this case. He told me the culprit was Softlayer, Bitly's hosting service, and that he would be providing more technical detail soon.

I checked with SoftLayer and the answer was on their Web site -- they block traffic from countries that are subject to U.S. trade and economic sanctions -- Cuba, Iran, North Korea, Sudan, and Syria. The rationalle for the SoftLayer policy is found in a Commerce Department guidance document.

So, we know what happened, but the real question is "Why now?"

Did Bitly know Cuba and the other sanctioned nations would be cut off when they moved to SoftLayer? (It looks like Bitly moved rather recently).

It turns out that SoftLayer began blocking Iran (and presumably the other countries) last February. Was that triggered by SoftLayer (or parent company IBM) lawyers exercising caution or were they pressured to change by government officials? Are they applying for an exception to the sanction?

Regardless, cutting Cuba off seems inconsistent with the policy of the current US administration. The Commerce Department page on the sanctions refers to "the President’s policy to chart a new course in bilateral relations with Cuba and to further engage and empower the Cuban people, announced on December 17, 2014."

This change inconvenienced a lot of Cubans -- does the US Government really want to do that at that time? Sanctions like this are a blunt instrument -- harming "good guys" like Cuba's new, Internet media as well as "bad guys."

This incident also reminds us of the fragility of Internet applications with dependencies -- the company or service your application depends upon can change its price or terms of use or just turn it off as in this case.

I'll see if I can get a better answer to the question why now? and will let you know what Doug's analysis reveals, but for now, we at least know what happened.

-----
Update 8/9/2016

I've asked IBM and SoftLayer why they made the decision to start blocking Cuba in February. IBM said they had no comment and SoftLayer did not return my phone call or email. I asked Amazon Web Services -- another cloud hosting company -- whether they blocked Cuban traffic and did not receive an answer to my email or phone message. (At least IBM had the courtesy of telling me "no comment").

Hitting that blank wall, I did a Google search and learned that:

• IBM acquired SoftLayer in 2013.
• In September 2015, the Treasury and Commerce Departments announced amendments to the Cuba sanctions regulations. "These regulatory changes build on the revisions implemented earlier this year and will further ease sanctions related to travel, telecommunications and internet-based services, business operations in Cuba, and remittances." The announcement states the desire to loosen sanctions on telecommunications & Internet-based services in order to enhance "the free flow of information to, from, and within Cuba, and better providing efficient and adequate telecommunications services between the United States and Cuba."
• In January 2016, Treasury and Commerce announced further amendments to the Cuba sanctions regulations. Treasury Secretary Lew said "We have been working to enable the free flow of information between Cubans and Americans" and the announcement goes on to say that Commerce’s Bureau of Industry and Security "will generally approve license applications for exports and reexports of telecommunications items that would improve communications to, from, and among the Cuban people."
• Shortly before President Obama's trip to Cuba in March 2016, a related announcement stated that "The Cuban assets control regulations currently authorize the importation of Cuban-origin mobile applications. The Office of Foreign Assets Control will expand this authorization to allow the importation of Cuban-origin software."

The administration has increasingly relaxed Cuban sanctions on telecommunication and Internet services. So, I am still wondering why, in February 2016, SoftLayer decided to start blocking Cuban traffic.

-----
Update 8/11/2016

Bitly CEO Mark Josephson has posted an article addressing the blocking of their site. He reiterates the fact that the decision to block Cuba was not their's and speaks in favor of an open Internet -- an "Internet you can see across," which has been a Bitly tagline since the company was founded.

He concludes that "We understand the rationale behind the rules in place from our partner and are working with them to change this. I’m confident that we’ll be able to address this with our partner, and if we can’t, we’ll try to find another way."

-----
Update 8/18/2016

Amazon Web Services and Rackspace allow Cuban traffic and Google Cloud Platform and IBM/SoftLayer block it. (All of them seem to be in similar businesses).

Iroko Alejo says Envato, Themeforest, Attlassian and Schema.org are also blocked.

Paypal allows remittances to Cuba, but they stopped a message accompanying a payment because it contained the word "Cuba." (I wonder what other words they filter for).

Reading the US policy statements on Cuban sanctions (above), it seems like the administration favors Internet communication with Cuba and would be unlikely to prosecute any of these companies for sanctions violations.

I wonder why some companies are more cautious than others.

-----
Update 8/27/2016

PayPal froze the account of Nathaniel Parish after he bought a Cuban cigar while he was in Mexico. In the article he summarizes the US administration policies on purchases of Cuban goods, which, like our communication policy, favors exchange.

Note that his purchase of Cuban cigars was legal.

-----
Update 8/29/2016

I have no way of estimating the number of Web sites blocked in Cuba, but we can get some idea of the number by looking at the numbers of Web sites hosted by known blockers Google and SoftLayer.

Builtwith.com tracks the number of sites served by hosting companies. Here are their current statistics for SoftLayer and Google:

As you see, Google hosts more than twice as many sites as SoftLayer, but SoftLayer hosts more of the top 10,000 sites than Google. (Follow the links above to interactive graphs of the history of these hosting platforms).

An interesting side note -- I was surprised at the amount of information Builtwith gathers on the hosted sites. For a fee, you can order a list of the sites hosted by a particular service along with the following data on each of their clients:

Domain name, Location on Site, Company, Vertical industry, Alexa and Quantcast statistics, phone numbers and email addresses, accounts on Twitter and nine other services, names, titles and email addresses of several employees and more.

For example, I could now give you information about 14 people who work for chronotrack.com.(but I won't).

Privacy is indeed dead.

-----
Update 10/14/2016

Cuban diplomat Josefina Vidal tweeted examples of over a dozen broken links to Softlayer and Google sites last month, followed by this rhetorical question:

I've spoken with people in the State Department and the Obama administration did not order those sites to be blocked and does not favor doing so. IBM and Google are independent companies acting on their own.

I first wrote about this blocking more than a month before Vidal's tweets -- I hope this post did not lead to her statement.

-----
Update 11/15/2016

I have heard from several sources that Bitly is working again in Cuba. I asked Bitly about it and they confirmed that the service is working again. They promised more news soon, but implied that their service is a special case and other Softlayer sites remain blocked.

IBM's SoftLayer cloud infrastructure service blocks Cuba -- why now?

UPDATED VERSION of this post.

Connection speeds from the LACNIC 25 Conference, hotels and the Google-Kcho Center

The LACNIC 25 Conference was held at the Havana Convention Center (Palacio de Convenciones) this week and Doug Madory, a frequent contributor to this blog, was there and sent me speed tests from the Convention Center and his hotel:

Convention Center speed test

Hotel speed test

Doug also forwarded a hotel speed test run by another attendee, Martin Hannigan, @TheIcelandGuy:

Hotel speed test

Access at the hotels was what we have come to expect -- around 1 mbps -- but connectivity at the Convention Center was much faster. I don't know whether that much bandwidth is always available at the Convention Center or if ETECSA allocated extra resources for this hi-tech conference.

Doug's next speed test was run at the Google-Kcho Center. As shown below, he used one of the center's chromebooks and connectivity was about four times the speed of the hotels.

Google-Kcho center speed test

Doug said there were maybe 4 or 5 chromebooks in use at the time he ran the test -- "Mostly young people on Facebook so not bandwidth intensive."

It is clear from this that the initial report that the center would provide connectivity for 40 simultaneous users at 70 times the speed of ETECSA's public hotspots, was incorrect. My guess is that they should have said 70 mbps backhaul shared among all the users currently online.

Finally, it is interesting to note that all four link speeds are roughly symmetric. We expect download speeds to be faster than upload speeds in the US because we download a lot more than we upload -- about 70% of our downstream, peak-period traffic is streaming entertainment.

But, in Cuba, home connections are over dial-up lines and the connections at public access spots and hotels are too slow for video entertainment. One mbps is fast enough for audio and even low-quality video chats, so symmetric bandwidth makes sense (as long as Cuba has El Paquete for the distribution of entertainment).

A second high-level US delegation to Havana to discuss telecommunication and the Internet

Daniel Sepulveda speaking at the University of Information Science (UCI)

A short post on the Web site of Cuba's Ministry of Exterior Relations reports that a high-level US delegation went to Cuba to discuss telecommunication during January 20-22.

The US delegation was led by Daniel Sepulveda, Deputy Assistant Secretary of State and U.S. Coordinator for International Communications and Information Policy and FCC Chairman Thomas Wheeler. They were accompanied by officials from the Departments of State, Commerce and Treasury, and the FCC. The delegation also included representatives of US telecommunication companies.

Deputy Minister of Communications, Jorge Luis Perdomo Di-Lella met the delegation along with officials from other ministries and representatives from business and academia. They visited the Joven Clubs, Polytechnic José Antonio Echeverría and the University of Information Science.

The post gives no substantive details on the meetings, but says they discussed the effect of the trade embargo and blocking of access to US Internet sites that were key to Cuba's scientific, technical and economic development. They also talked of the scope and limitations of the new regulations adopted by the US government on Cuban telecommunication, which probably means that Chairman Wheeler clarified the implications of the recent removal of Cuba from the FCC "exclusion list" (which is now empty). That sounds like a quick summary of issues the diplomats and officials may have raised, but there was no mention of which US business leaders were in the delegation or what they may have said.

This was a follow-on to a similar meeting held last March. Based on the minimal reporting of the two meetings, the only difference would seem to be the participation of the FCC.

In addition to clarifying the implications of Cuba being removed from the exclusion list, the Cubans and FCC staff may have discussed alternative infrastructure ownership and regulation policies. Since Cuba is late to the Internet game, they are free to consider the wide variety of policy alternatives adopted by different nations. It may be unlikely, but it is possible that in doing so they could come up with a uniquely Cuban Internet.

-----
Update 1/26/2016

Daniel Sepulveda, who has led both US government delegations to Cuba, has given interviews on the trip to the Miami Herald and OnCuba magazine (in Spanish). You should read both interviews, but I will summarize some of the things that jumped out at me.

Sepulveda said there are at least a half-dozen proposals — from US and non-US companies — to construct a north-south undersea cable between the US and Cuba. An undersea cable connecting Havana and Florida would provide backup for the ALBA-1 Venezuelan cable, add capacity and reduce latency. Perhaps more important, it would reduce the load on Cuba's domestic backbone. This is something ETECSA can and should be negotiating on behalf of the Cuba people, even if it requires government subsidy to attract capital, and they have requested specific, written proposals.

Sepulveda also pointed out that such a cable would establish both a psychological and physical connection between the two countries -- a sign of healing.

The ball is now in Cuba's court. In the past, the embargo limited, but did not stop the Cuban Internet. Mexico's Grupo Domos and the Italian phone company STET were investors in ETECSA, US equipment was available through third parties and China has provided the undersea cable and much domestic infrastructure. Sepulveda pointed out that there are no longer restrictions on US telecom company dealings with ETECSA or other Cuban organizations. (private programmers can also work for US companies).

Sepulveda feels a sense of urgency -- neither he nor President Obama will be in the government next year, and, while he does not believe it will be possible for the next administration to reverse the advances that have been made, it will be possible to delay implementation and stop further progress. There is an issue of trust in both directions -- trust that the US and the Internet will not undermine the Cuban government and trust that the Cuban government will be open to foreign investment and will not constrain investors with overly burdensome regulation. US companies need positive signs from Cuba if they are going to invest.

The US delegation included Dean Garfield, president of the Information Technology Industry Council, and representatives from Cisco Systems, Comcast, the North American division of Ericsson, a Swedish communications company, and other government and industry officials. Google, which has expressed considerable interest in Cuba, was conspicuously absent -- perhaps due to the lack of trust Sepulveda referred to.

Cisco has proposed establishing a Cisco Academy training and certification program at the University of Information Science. That is the most concrete proposal I have heard of and, if it is approved, it would signal Cuba being open to competition for Huawei, which has a dominant position today. Opening a Cisco Academy at a major computer science university would both give Cisco a foothold in the Cuban infrastructure market and signal Cuban willingness to have infrastructure competition.

Sepulveda favors rapid rollout of fourth generation mobile connectivity, mentioning Vietnam, Myanmar, Ecuador, Bolivia and the Dominican Republic as examples. That must have brought a smile to the face of the representative from Ericsson. (Doug Madory has suggested Myanmar as a model).

Finally, the delegation met with independent bloggers -- I wonder which ones and what was said.

Those are a few highlights, but the interviews say more.

Here is a short video of Sepulveda speaking with the Miami Herald:

Daniel Sepulveda during his OnCuba interview

-----
Update 1/28/2016

I mentioned above that the US delegation met with several Cuban bloggers and entrepreneurs, and I wondered who they had met with and what was said. Two of the bloggers they met with were Norges Rodríguez and Taylor Torres and Norges has summarized the meeting in a blog post.

After summarizing the meeting, he inlcuded the full text of the interview of Sepulveda, which I highlighted above, and he promises to post more on their wide-ranging meeting.

I do not know which other bloggers and entrepreneurs the delegation met with, but the inclusion of Taylor and Norges is interesting because one blogs on art and culture, the other on telecommunication technology and policy. We tend to focus on topics like infrastructure, access to information, social networking, etc., but the Cuban creative community has the potential to become a rich source of Spanish language entertainment and art content. (Netflix came to Cuba less than two months after the December 17 opening of relations -- I suspect in search of potential content as well as subscribers).

It is also encouraging that high-level US officials and business people are meeting with bloggers and entrepreneurs as well as government officials and ETECSA executives. Perhaps we will see a uniquely Cuban approach to the Internet.

Norges Rodriguez, Taylor Torres, Tom Wheeler and Daniel Sepulveda

Image from Taylor Torres' art-focused blog

-----
Update 1/29/2016

FCC Chairman Thomas Wheeler has written a blog post on the trip to Cuba. I don't think he adds much of substance, but it is good to see Cuba on his mind and concludes that he came away from the trip with "a newfound understanding of both the opportunities and challenges facing Cuba in terms of communications technology and access."

Perhaps the most interesting aspect of his participation in the trip was that he met with students and bloggers as well as government officials and executives of state enterprises. Here he is shown with the Cuban bloggers shown above.

Criticism in the US and Cuba

Criticism has the possibility of being more effective in Cuba than in the US.

Doug Madory, Director of Internet Analysis at Dyn Research, sent me a note saying the Cuban blog Internet en Cuba was down, displaying the message "This site has been archived or suspended." He tried emailing the blog's author, but that email bounced.

You cannot see the blog at this time, but the Internet Archive has stored a couple of recent posts, including this one:

The post is critical of ETECSA for not being transparent about the cause of an outage and whether they plan to compensate users. I have no way of knowing whether the problems with the blog are related to this criticism or not -- I hope not -- but it got me thinking about criticism in Cuba versus the US.

I have been quite critical of my Internet service provider, Time Warner Cable, in blog posts. For example, I have said they violate network neutrality, offer terrible customer service, abuse their monopoly power and mislead customers on pricing. This is the image I used to illustrate the post on misleading prices:

In spite of all that, I continue to receive my usual, overpriced service.

In the US, we are generally free to criticize ISPs, political candidates, corporations, the government, etc., but that criticism has little effect. My opinion of Time Warner Cable is common and many people have pointed out the same failings as I have, but nothing has changed.

Cuban blogger Carlos Alberto Pérez has said "I don't criticize to knock the system down. On the contrary, I criticize to perfect the system." I may be naive (probably am), but criticism has the possibility of being more effective in Cuba than in the US.

-----
update 1/27/2016

The blog has suddenly reappeared with two new posts and email to the author, who uses a pseudonym, is working again. No explanation of its absence was given.

Year end interview of the president of ETECSA

"The people want to be connected."

ETECSA president, ingeniera Mayra Arevich Marín

Mayra Arevich Marín has been president of ETECSA for four years. The following are a few points from a recent year-end interview.

• Internet access was improved through the rollout of Nauta rooms, WiFi hotpsots and improved connectivity at institutions that are important to the society.
• By the end of the year, there will be 65 WiFi hotspots and they will add 80 more during 2016.
• Today there are over 700 public access points in navigation rooms, cyber-cafes, hotels and airports.
• Average daily access is over 150,000 people -- double last year.
• They are encouraging the move to permanent Nauta accounts and hiring agents at WiFi hotspots to stop resellers. They are also experimenting with having people at the WiFi hotspots to assist customers. (It takes time to train support and marketing people).
• They are also working on a system to let people buy time online rather than through an agent. (It seems they could have done this from the start -- send a 2 CUC text message to ETECSA in return for a 1-hour passcode).
• They are working on infrastructure to support this access. They have expanded the capacity of their existing data center and will build two new data centers and augment backbone access to the international undersea cable in 2016. (She did not mention it, but the bulk of Cuba's international traffic shifted from satellite to cable this year, enabling the increase in access).

In addition to access, she mentioned new applications and improved connectivity in several government sectors:

• There are now 40 thousand doctors who connect from their homes to the Internet via Infomed. They also improved the connectivity of health institutions.
• The Ministry of Justice is putting applications like access to municipal records online.
• Fiber connectivity has been provided at over 25 higher education facilities. By the end of the year, all Cuban universities were connected.
• An interbank network was created and banking applications implemented. There are now 773 ATMs in Cuba, 150 of which were installed this year.
• The Attorney General's office, the Cuban Institute of Friendship with the Peoples (ICAP), the Ministry of Agriculture, the National Institute of Water Resources and BioCubaFarma business group have improved fiber connectivity.
• She said they are preparing new service offerings for 2016, but did not say what they were.

Entertainment is one critical application that was not mentioned. Today digital entertainment is being handled off line by El Paquete, but normalization of relations with the US will at some point eliminate the piracy subsidy upon which it is based, leaving a cost gap.

Finally, Arevich Marín said that since they must pay for infrastructure and equipment with convertible currency, they need to continue generating revenue through expensive service, foreign recharging, exportable services, international voice and roaming charges and government subsidy.

In a way, this was a typical year-end summary by any CEO -- mentioning achievements for the year past and hinting at some plans for the coming year, while ignoring problems.

Viewed from the perspective of the Internet in a developed nation, I am saddened by how little connectivity Cubans have, but I am more interested in where Cuba will be five or more years from now, so, for me, the key point in this interview was the last one -- citing the need for convertible currency. It is an indication that, at least for now, Cuba has decided to be relatively self-sufficient with respect to the Internet, but can they afford a self-sufficient Internet?

The conventional wisdom is that if Cuba wants to expand the Internet quickly, they should privatize and regulate the Internet in return for foreign investment. For example, Doug Madory has suggested licensing mobile providers, an approach that has led to rapid improvement of the mobile Internet in Myanmar, another "green field" nation. Cuba is seeking foreign investment in industries like mining and oil production, but the Internet is basic domestic infrastructure that might reasonably be kept independent. They should consider alternatives for infrastructure ownership and regulation along with foreign investment.

-----
Update 2/5/2016

ETECSA officials held a press conference yesterday. Here is some of what they said. (My comments are in parenthesis).

At the end of the year there were 3.3 million mobile accounts. (Mobile Internet access is primarily used for personal communication and entertainment, not content creation or productivity applications).

They acknowledge and are working on peak load problems.

Rates have been reduced. (But they remain high enough to create a digital divide within Cuba).

They cautioned that the Old Havana pilot study was only a trial.

There are now agents selling telecommunication cards and recharge coupons (but not satellite access, which US operators are now allowed to provide).

They acknowledged that some of the public access hotspots were in inappropriate locations.

100 cellular base stations will be upgraded from 2G to 3G during the first half of 2016. (How many base stations are there altogether, what percent of the population will have 3G coverage at their homes and offices and why 3G)?

They will establish 80 new public WiFi hotspots this year and offer a variety of handsets for sale. (Is ETECSA the sole vendor for handsets)?

-----
Update 3/31/2016

ETECSA executives held a round-table discussion on Internet services and WiFi hotspots and you can read the transcript here.

A couple of items caught my eye:

• They will be upgrading access points to accommodate both 2.4 and 5 Ghz connections since 82 percent of today's connections are at 2.4 Ghz and today's laptops and tablets can utilize the 5 Ghz band.
• Approximately 200,000 users connect daily at WiFi hotspots -- up from 150,000 reported in December.
• They plan to upgrade 100 radio base stations (RBS) in Havana to 3G mobile. (Note that "RBS" is a term used by Ericsson, not Huawei).
• Details of what will be a very expensive home connectivity project have not been ironed out, but they plan to upgrade equipment to provide a capacity of 1.2 million ADSL land lines. (That is fewer than the goal of nearly 2 million private homes that was stated in a home-connectivity presentation that was leaked last year).

They said a lot more, so you might want to read the transcript for yourself.

In general, I was struck by their reiteration of commitment to already obsolete equipment like ADSL to homes and 3G mobile. I hope they consider this and the WiFi hotspots stopgap technology and are making plans for leapfrogging today's technology and today's infrastructure ownership and regulation policies.


-----
Update 8/16/2016

Last month, vice minister of communications Wilfredo González Vidal reportedly announced that they planned to have 655 public access locations in ETECSA "navigation rooms, Youth Computer Clubs, hotel and airports by the end of this year. He also said they now have 125 WiFi hotspots.

But, at the end of last year, Mayra Arevich Marín, president of ETECSA said there were already 700 public access points in navigation rooms, cyber-cafes, hotels and airports (see above).

Perhaps I misunderstood -- my Spanish is bad at best -- maybe the vice minister was saying they would add 655 new public access points. Here is the quote: "Los planes para ampliar el acceso público a internet prevén el aumento de capacidades con un total de 655 instalaciones a finales de este año" (my italics).

Regardless, proud announcements of such low numbers are discouraging. It is reminiscent the highly publicized offerings of the artist Kcho. What he did was fine, but it is an inconsequential drop in the bucket in a nation of over 11 million people.

By the end of the year, two years will have passed since the beginning of rapprochement between the US and Cuba. The Internet related announcements made at the time of President Obama's visit to Cuba have turned out to have been little more than public relations. Citmatel is embarrassingly out of touch with their goofy software and content offerings. Etc.

If I saw evidence of serious long range planning for "leapfrogging" today's technology and policy, I would understand a slow, stop-gap approach to public access, but I don't.

-----
Update 8/22/2016

I noted inconsistency (above) in the number of public access points claimed by the Cuban Ministry of Communication and ETECSA, so I checked the ETECSA connectivity page for clarification. They list 259 public access rooms with a total of 921 computers, as follows:

ETECSA and the Ministry of Communication seem to agree that there will be at least 655 public access spots in ETECSA navigation rooms, Youth Computer Clubs, hotels and airports by the end of this year. I don't know how many are in hotels and airports, but the navigation rooms and Youth Computer Club installations shown here are well below that target.

The Ministry of Communication also said they now have 125 WiFi hotspots, but ETECSA lists 178. (They are silent on the characteristics of each -- for example, on whether the backhaul capacity from each is the same).

Some time ago, I noted that the roles of ETECSA and the Ministry of Communication with respect to the Internet are unclear. So are the statistics.

-----
Update 9/11/2-16

Juventad Rebelde reports that there are now 1,006 public access locations in Cuba -- 200 WiFi hotspots, 193 “navigation rooms” and 613 other locations at Youth Computer Clubs, hotels, Health Ministry and Postal offices and airports. The article also says there are 250,000 WiFi connections daily -- 80% at 2.4Ghz.

ETECSA lists the locations of the WiFi hotspots here and the following map shows the number of WiFi hotspots, navigation rooms and other access locations in each province.

-----
Update 9/23/2016

ETECSA has plans to install WiFi hotspots along a five mile stretch of the Malecon by the end of this year. They did not release details on the number of access points or their distribution or backhaul plans, but it will be a really nice touch if all goes well.

Cuban Internet interruptions

Is there a connection between Cuba's email interruption and access to the Reflejos blogs?

Cuban email has been interrupted and colleague Doug Madory discovered that access to Cuba's Reflejos blogs at is inconsistent. One can access the Reflejos home page, but some links work and others do not. For example, try these permalinks:

http://internet1.cubava.cu/2015/11/19/internet-en-cuba-porque-estamos-desconectados/

http://alejandro13.cubava.cu/2015/10/14/he-sido-hackeado-el-proyecto-que-promete-conocer-si-tu-email-se-ha-visto-comprometido/

or the blog home pages: http://internet1.cubava.cu and http://alejandro13.cubava.cu

The connections to these requests are inconsistent. Sometimes they work and others not. When they fail, they do not time out, they just take forever, so a handshake must at least be made.

Does that suggest some sort of flooding DoS attack? Do these problems have a common cause?

-----
Update 11/21/2015

The Reflejos blogs seem to be working again, but I've not seen an update on Nauta email.

-----
Update 12/1/2015

I've been told that Nauta email is only working on the WiFi network from 7am to 11pm and it will not work in the Nauta rooms.

Before and after Cuba's shift to the ALBA 1 undersea cable

Northwestern researchers Zachary S. Bischof, John P. Rula and Fabian E. Bustamante have published In and out of Cuba, a paper characterizing Cuba's connectivity.

They gathered data during March and April 2015 and found that traffic going out of Cuba typically traveled through the ALBA-1 cable, but traffic coming into Cuba was often routed over satellite links, adding about 200 ms to round trip times (RTTs).

The data was gathered using RIPE Atlas probes, small, USB-powered hardware devices that hosts attach to an Ethernet port on their router. There are 8,771 of these probes around the world, including one in Havana. Upstream results from that probe may or may not have been representative of the nation as a whole.

While the study results were accurate last Spring, they do not reflect the current situation. Last July, Doug Madory, Director of Internet Analysis at Dyn Research alerted me to the fact that Cuba's international traffic had largely shifted to the undersea cable.

I forwarded the Northwestern paper to Doug and he sent me the following graphs, showing traffic to both of Cuba's autonomous network operators for March 30, 2015 and October 30, 2015:

In March, there was considerable satellite traffic from NewCom and Intelsat.

By October, there was little satellite traffic.

As mentioned above, the Atlas probe may not have been representative. The graph on the left side of the following figure shows traffic to the IP address range the Atlas probe resides in. During the time the Northwestern team gathered their data, inbound traffic was almost entirely via satellite (Intelsat or NewCom), but it shifted subsequently. The graph on the right shows that the majority of traffic coming to representative destinations of the Cuban ISP ETECSA was routed over the cable during the study period. In July, it nearly all shifted to the cable, where it is today.

Most traffic to the probe shifted to cable in June (left). ETECSA received relatively
little satellite traffic during the study period and essentially none after June (right).

The analysis by the Northwestern team is thorough and insightful, but it seems they have been caught by academic publishing delays. They say this was just the start of ongoing study of the Cuban Internet and I am looking forward to seeing their future work.

Finally, today's Internet access and speeds in Cuba are very poor but, given today's low Internet penetration, domestic infrastructure, not international capacity is the key constraint.

Cuban infrastructure investment -- China won the first round

China won the first round, what about the future?

In December 2014, the administration announced that we were taking "historic steps to chart a new course in our relations with Cuba and to further engage and empower the Cuban people." The following month, the US International Trade Commission began a study of the economic effects of US restrictions on trade with and travel to Cuba. They held hearings on potential exports in several sectors and I testified on potential telecommunication exports.

In March, the US sent a high-level delegation to Cuba to discuss telecommunication and the Internet and no doubt Internet service and equipment companies began analyzing the potential Cuban market. Most visibly, Google visited several times and eventually made a concrete proposal for the installation of some sort of wireless infrastructure, but that offer was rejected, perhaps for lack of trust in the US Government and Google.

Google made several trips to Cuba, but their proposal was rejected.

This month the White House extended our policy, authorizing US companies to establish a business presence in Cuba and provide "certain" telecommunications and Internet-based services or do joint ventures or enter into licensing agreements to market such services.

To date, this effort has led just a few small Internet deals like Netflix offering Cubans accounts, Airbnb renting rooms or Verizon offering cell-phone roaming in Cuba.

Cuba has turned to China, not the US, for Internet connectivity and equipment and is committed to doing so in the short term future.

China played a major role in the financing and construction of the ALBA-1 undersea cable, which connects Cuba to Venezuela and Jamaica. It was reported that China lent Venezuela $70 million to finance the cable, which was installed by a joint venture made up of Alcatel-Lucent Shanghai Bell and Telecomunicaciones Gran Caribe (TGC) -- TGC is a joint venture between Telecom Venezuela (60%) and Cuban Transbit SA (40%), both state-owned companies.

The cable landed in Cuba in February 2011, but the first traffic was not transmitted until January 2013. Much of Cuba's international traffic continued to be routed over satellite links until July 2015, when nearly all of it had finally shifted to the cable. Cuba's international traffic continued to be routed over slow, expensive satellite links for over four years because the cable landing point is at the east end of the island and there was little domestic infrastructure to connect it to Havana and other locations.

The ALBA-1 cable traffic has shifted from satellite (blue) to cable.

At the time of the cable installation, we speculated that China might play a role in building the domestic infrastructure needed to reach it and it turns out that Cuba had awarded Huawei a contract to build a national fiber-optic network in the year 2000. Today there is a backbone network connecting the Cuban provinces to the cable landing point. The current load is light compared to expected future traffic from homes, schools, universities and public access locations, so Cuba must be planning a faster, more comprehensive backbone and I imagine Huawei is involved.

ETECSA backbone diagram, date/status unknown, source: Nearshore America

Huawei equipment was also used in the recent installation of 35 WiFi hotspots across the island. Since they claim the access points will support 50-100 simultaneous users at 1 Mb/s speed, these 35 locations must connect to the national backbone network. While 35 access points are a drop in the bucket, Cuba is committed to adding more. Counting WiFi, "navigation rooms," Youth Clubs and hotels, there are now 683 public access points in Cuba, all of which reach the backbone.

Huawei WiFi antennae

In addition to expanding public access and the backbone, they plan to make DSL connectivity available to 50% of Cuban homes by 2020. (Note that that is not to say 50% of Cuban homes will be online). Doing so will require new equipment in the telephone central offices serving those homes and Huawei will supply that equipment. Two other Chinese companies, ZTE and TP Link are providing DSL modems for network users. (ZTE has an office in Havana and may also be involved in the backbone network).

Home Internet: Huawei central office equipment and ZTE and TP Link modems

Cuba also has plans to connect all schools and make fiber connections to the backbone available to all universities. I don't know whose equipment will be used for those upgrades, but, if Huawei is the backbone vendor, I suspect that they would have the inside track on customer premises equipment (CPE). A recent market research report shows that Chinese CPE sales are growing rapidly, fueled by a large domestic market.

Lina Pedraza Rodríguez, Minister of Finance and Prices, said that Cuba is in
"very advanced" negotiations with Huawei, May 2015.

In spite of China's success in Cuba, all has not been perfect. As this Wikileaks memo from the US Interests Section in Havana shows, the Chinese have had some difficulty collecting Cuban debt. Cuba remains a tricky place to do business.

Finally, note that all of these sales are for equipment, not network operation. While Huawei has sold Cuba equipment, the backbone installation has been supervised by a Cuban engineer who has worked for Huawei since 2002 and Huawei does not seem to have an office in Cuba. Cuba bought Telecom Italia's share of ETECSA, Cuba's monopoly telecommunication company, in 2011 and remains independent. That may turn out to be a good or bad thing for the Cuban people, depending upon ETECSA's policy and goals.

It looks like China has won the first round. That's the bad news for US companies. The good news is that very little infrastructure has been sold so far and much of what has been sold and is planned for the near future is already obsolete by today's standards. That says there will be a much larger second round -- will the US be a player?

-----
Update 10/1/2015

A Cuban reader commented on Huawei's success in Cuba:

In early 2000 gradually they replaced all Cisco router by Huawei, including my office. probably today 95% of all routers in Cuba are Huawei, a "legacy" of Ramiro Valdez was minister, and the millions that Cuba spent in the "battle of ideas."

He also said that Huawei has had an office in the Miramar Trade Center in Havana for over ten years. (I could not find it using Google).

Note that Ramiro Valdés called the Internet "the wild stallion of the new technologies," which "could and should be controlled and used to serve peace and development" in spite of the fact that it construes one of the "mechanisms for global extermination."

If dealing with Huawei was politically motivated, the Internet infrastructure market may open in the future. (Some readers will claim that payoffs were involved, but I have no evidence one way or the other on that).

-----
Update 3/5/2016

The home page of the Mariel port web site features images of Chinese cranes, but China has supplied more than cranes. Doug Madory, Director of Internet Access at Dyn Research, sent me an email saying he had found evidence of Chinese software on the site -- an error message with Chinese characters.

I went to the site and, at first, could not reproduce the error message, but I have been able to subsequently-- it is due to slow load time. When the site begins to load, it displays three links as shown below, but the page is not fully loaded. Clicking on "Sobre TC Mariel" displays the error message with Chinese characters ("Data failed to load!" according to Google Translate), shown below.

Chinese error message on the Mariel site

Doug followed up with a snippet of JavaScript with Chinese comments, saying things like "Cookie's name," "Will be saved" and "Cookie format is a semicolon spaces." (It's kind of cool to see that Chinese programmers write comments just like we do).

Chinese comments in Mariel site code

It will not be easy for the US to compete against the Chinese in Cuba -- Huawei and others are well-established, formidable competitors. That is one of the hidden costs of the Cuban embargo.

=====
Update 6/3/2916

Is this a coincidence? The Chinese news agency, Xinhua, just posted a rosy puff-piece about Cuba's plan to "expand Internet access in public areas and private businesses." It is not a news report -- it rehashes things like their planning to deploy more public access via WiFi and Internet cafes and takes a shot at the US trade embargo.

A couple days later, the Wall Street Journal reported that "The U.S. Commerce Department has subpoenaed Huawei Technologies Co., demanding that the Chinese telecommunications giant submit all information on its export and re-export of technological goods to Iran, North Korea and other sanctioned nations."

Can the US really do much about Huawei in Cuba? I doubt that they are highly dependent upon components from US suppliers and, to the extent that they do use US components, wouldn't they find substitutes if blocked?

=====
Update 12/27/2016

China has been Cuba's most important source of Internet consumer and infrastructure equipment and now China's Haier Group will be manufacturing low-end laptops and tablets in China. Haier is a large, multinational manufacturer of home appliances and consumer electronics -- they are in 29 countries and recently acquired General Electric's appliance division. For now, they will be making low-end laptops and tablets in collaboration with GEDEME, a Cuban manufacturer, but, what are their plans for Cuba? Is this a strategic investment?

GEDEME factory -- four managers, one worker, no robots -- symbolic if not literal.

-----
Update 5/18/2017

David Crosswell points out that China has been active in other parts of the Cuban economy as have the Russians and Iranians and provides links to several examples in his comments on this post.

Cuba, given its proximity, is critical to our national security -- a lesson we learned during the Cuban missile crisis. Trump should think about "unvetted" Iranians 90 miles from Florida.

https://anewdomain.net/china-wins-first-round-cuban-internet-investment-analysis/

-----

Update 4/1/2025

It was just pointed out to me that a version of this post was cited in the recommendation against the connection of the ARCOS-1 cable, which I supported.

Cuban international traffic shifts from satellite to the ALBA-1 undersea cable

Doug Madory, Director of Internet Analysis at Dyn Research, sent me a note on Cuba's international traffic. As you see here, on July 1, nearly all satellite traffic (blue and green) was re-routed to the ALBA-1 undersea cable:

As a result, median latency has stabilized at around 210 milliseconds:

This is good news for Cubans who have Internet access at work, school or ETECSA hotspots and navigation rooms.

There must be relatively fast terrestrial connectivity to the cable landing point at Siboney Beach. Does anyone have any information about the nature of that connectivity? Huawei is installing home DSL and WiFi -- have they also installed an inter-province backbone? Could there have been an unannounced deal with medium-earth orbit satellite provider O3b Networks?

-----
Update 7/20/2015

Huawei may be installing home DSL and WiFi hotspots in Cuba, but Doug Madory has discovererd at least one piece of Cisco equipment -- a 2800 router at the University of Havana. (I'd be curious to know how they obtained it.)

I am not familiar with the Cisco 2800, so I Googled it to get the specs. I was saddened to see that it is old equipment, near the end of its support life -- the end date for software maintenance has already passed and hardware support will end soon.

Yoani Sánchez' digital newspaper, 14ymedio, is online

The daily Web newspaper 14ymedio looks like a fully traditional digital "newspaper" with news, sports, culture, opinion, fashion tips, weather, etc.

The 14ymedio "declaration of intentions" says the team is committed to promoting truth, freedom and human rights, without ideological or partisan ties -- they hope to provide a space for respectful debate and to contribute to the peaceful transition to democracy. A group of 28 writers and intellectuals, including Mario Vargas Llosa and Lech Walesa, signed a request that the Cuban government respect 14ymedio and allow it to exist with free expression.

Evidently some pro-government people were not happy with the launch of 14ymedio -- the Cuban domain name server was hacked, and Cubans looking for 14ymedio.com were automatically redirected to a pro-government Web site. Doug Madory of Renesys reported that the hack was local to Cuba and the site was visible in foreign nations. Madory speculated that the hack had been done by someone inside ETECSA.

Today, relatively few people on the island can afford to access 14ymedio online, but stripped down PDF and text versions are available. For the first issue, they were quite minimal -- hopefully they will improve -- becoming something like the daily New York Times Digest. It could also be included on flash drives that are regularly distributed on the island. Regardless, I hope the hack was a rogue action and the paper turns out to be the impartial forum envisioned by the founders.

Has the ALBA-1 undersea cable changed anything?

In January, I posted a note about a blog post by Renesys analyst Doug Madory showing that Cuba had begun testing the ALBA-1 undersea cable. Madory's data shows the round trip "ping" times for packets traveling between Cuba and four other cities.

The slow speed data points (A) are using the old satellite links.  The medium speed links (B) are asymmetric -- outbound via satellite and return through the cable.  The high speed links (C) imply two way cable traffic.

A number of people commented on Madory's post, some indicating that they had indeed noticed faster speed in accessing Cuban servers, but, consistent with the Renesys data, others did not.

Madory updated the data in March. As we see here, the majority, though not 100 percent, of Telefonica traffic is being carried over the cable.

Regardless, if Cuba lacks the political will and domestic infrastructure to connect users to the cable, it will have little practical effect.  The Cuban government has said the first applications of the cable would be those that benefit society.  It seems to me that university and medical connectivity would fill that bill and be low-hanging fruit.

It has been three months since Madory first detected traffic.  Has anyone noted any improvement in their service?

First traffic on the ALBA-1 cable

Doug Madory, who has been keeping us up to date on traffic (or the or lack of it) on the ALBA-1 submarine cable between Venezuela and Cuba pointed me to a new blog post this morning, in which he reports limited cable traffic.

For the past six years, three satellite providers, Tata, NewCom and Intelsat have served Cuba. But, as you see in the above graph (click to enlarge), Telefonica traffic (dark grey) began last week.

Madory also monitored the round trip time to send data packets from Guadalajara, Mexico, Dallas, Texas and Sao Paulo and Joao Pessoa, Brazil to Cuba. He noted a significant speed up on all four routes at the same time on January 14th, indicating that some Telefonica traffic is being carried over the cable. But, since the average time remains quite high, Madory concludes that

Telefonica's service to ETECSA is, either by design or misconfiguration, using its new cable asymmetrically (i.e., for traffic in only one direction), similar to the situation we observed in Lebanon in 2011. In such a configuration, ETECSA enjoys greater bandwidth and lower latencies (along the submarine cable) when receiving Internet traffic but continues to use satellite services for sending traffic.

He goes on to speculate that the first evidence of ALBA-1 traffic and the elimination of exit visas might be part of a greater trend towards a freer and more open Cuba.

Hard data on the idle ALBA-1 undersea cable

I received an email today from Doug Madory of Renesys, a company that monitors the dynamic state of the global Internet.

Madory wrote that there is no evidence of a submarine cable in use in Cuba in 2012. He said that latencies to Cuba are very stable and clearly satellite (>480 ms).  He attached the following visualizations (click to enlarge):

The numbers in the figure legends indicate the connecting autonomous networks -- CubaData (11960) is the state telecom of Cuba, and they have three satellite providers Tata (6453), Intelsat (22351) and NewCom (32034).

Renesys is "The Internet Intelligence Authority" -- they constantly monitor the state of the global Internet. You may have seen their reports of network outages when nations went off line during the Arab Spring, for example, this Syrian episode. You can get a sense of what they do by following their blog and Internet events bulletin.