Tuesday, April 18, 2017

No longer a cord-cutter -- I've spliced the cord

My 2014 post "How I cut my Time Warner bill by 33%" has been viewed 152,846 times -- the most of any in the history of this blog. The bill-cutting technique is simple -- threaten to cancel your service and the ISP will renegotiate the price.

I recently repeated the process, with a twist.

I was an early cord-cutter -- getting my local TV with a rabbit ears antenna and streaming the rest from the Internet. That worked fairly well, but I could not get local content in some of the rooms of my house and even in the best room, there would be an occasional glitch and I had to play around with the antenna orientation. I tried amplified antennas, but none were better than my rabbit ears and I am too lazy to install a rooftop antenna. (The local TV transmitters are on a mountain 24.5 miles as the crow flies from my home).

My monopoly ISP bill crept up over time, as monopoly ISP bills do, and my old monopoly ISP, time-Warner Cable (TWC), had sold to a new monopoly ISP, Spectrum.

Spectrum started sending out flyers offering good deals to new subscribers -- Internet, phone and cable-TV service for a little less than I had been paying TWC. I called and offered to switch to the introductory offer and they accepted -- I spliced the cord.

I now get rock-solid local TV and a DVR for less than I was paying before. That is an improvement, but nothing like I could get by moving to place with a competitve Internet service market like Riga, Stockholm or Korea.

Are you hoping new wireless technology like 5G mobile or PCell technology from Google will provide ISP competition? The technology remains to be seen in the field but, if it turns out to be a threat, the ISPs will work hard to fight competition, for example, by outlawing the sharing of public infrastructure.

In spite of periodic renegotiation with my ISP, the cost is drifting up and I pay for streaming services like Netflix and Amazon Prime, but I seldom go out to a movie these days. It looks like the long-run losers will be movie theaters and the public.

Thursday, April 13, 2017

Presentation on the implications of the Internet for politics

I teach a class on the applications, implications and technology of the Internet and we look at relevant current events each week. Last semester there were many current events dealing with the election and I accumulated a large, chronologically ordered, PowerPoint slide deck on the political implications of the Internet.

Last week I substituted for a faculty colleague and gave two 75-minute lectures on the topic, using selected slides from the full deck. The selected slides are not chronological, but organized as follows:

  • Historical context
  • Lying
  • Fact checking
  • Fake news for money
  • Fake news for politics
  • Fake images
  • Trump dominated social media
  • More historical context - disillusion
  • Real world consequences
  • Hacking
  • The Internet is ephemeral
  • Breitbart – “alt right” press
  • Money behind the scenes
  • (Imperfect) fixes
  • Future fake media
There are over 100 selected slides with mnemonic images, few words, links to supporting material and notes. (I use annotated slides in lieu of a textbook). Here are thumbnails of a few of them:

Saturday, March 04, 2017

Why did't the Internet zap Singapore's Straits Times newspaper?

A Wednesday edition of the Straits Times had 16 pages of color classified ads in spite of Craigslist.

Business Insider
US papers employed 56,900 full-time journalists in 1990, the year Tim Berners Lee began testing his World Wide Web software, and they employed 32,900 in 2015. The disruption of the newspaper business began 22 years ago, when Craig Newmark launched his classified ad site, Craigslist. (Note that Newmark now generously supports investigative journalism and fact-checking organizations). Newspapers have adapted to the Internet by adding digital editions, but they generate less ad revenue than print editions have lost.

Thomas Jefferson and a lot of other smart people believed that democracy requires a free press. (See these quotes). If we agree with Jefferson, et al, that investigative journalism and fact-checking are important facilitators of democracy, can the Internet at least help keep organizations like newspapers alive?

At least one newspaper seems to be OK -- can we learn from it?

I was in Singapore a few weeks ago and picked up a copy of the 2/1/17 edition their major, English language newspaper, the Straits Times. I was impressed -- the paper was physically large, every page had color and the price was only S1.1, about 78 US cents. When I got home, I compared it to a 2/22/17 copy of my home town newspaper, the Los Angeles times, which sells for $2. (Both were Wednesday editions).

Number of pages in each section
The pages of the Strait's Times were 27 percent larger than those of the LA Times (which shrunk after it was purchased by Tribune Publishing in 2000) and there were more of them, as you see here. And what about those "dead" classified ads? The Straits Times had 16 pages of classifieds and the LA Times only 2/3 of a page at the end of the Sports section.

Why does the newspaper business in Singapore seem to be thriving, while US newspapers are having a hard time?

It's not the market size. The population of Singapore is about 5.6 million, the poulation of Los Angles is about 4 million and greater Los Angeles is about 10.2 million.

It's not economies of scale. In August 2016, the Straits Times had a daily print circulation of 277,100 and 116,200 digital. The LA Times media kit says their weekday circulation is 690,870 and it's 955,319 on Sunday.

The Straits Times is not a local paper -- they have 16 bureaus and special correspondents in major cities worldwide. (Both of the stories that were "above the fold" on the front page of the edition I picked up were about US politics).

Maybe there is no Craigslist in Singapore -- but there is.

The government role

Singapore's fast, affordable Internet connectivity makes the digital edition of the Straits Times attractive. There are five competing ISPs and most of the country is covered by fiber as well as copper. A 1 gb/s account will set you back S$49.99 per month if you sign a two year contract or S$59.99 without a contract. For two gig, you pay $69.99 with a two year contract. The slowest offering is 100 mb/s. (Singapore dollars are around 71 US cents).

The Singapore government deserves a good deal of credit for their Internet service. In 2000, I worked on a study of the Singapore Internet and, with the help of my nephew who was with Goldman Sachs in Singapore, developed this figure:

Singapore, Inc.

As you see, the government had equity positions in the ISPs and an indirect link to Singapore Press Holdings, a media conglomerate that owns the Straits Times. The government provides wholesale backbone connectivity to those competing retail ISPs. (Other cities, notably Stockholm, have followed a similar strategy and Google has done so in Africa).

Competition is the key to the success of the Internet in Singapore and, while the current US administration claims to like free markets, moves to weaken net neutrality, set-top box standards and municpal wholesale networks strike me as anti-competitive. (Also, see this interview of outgoing FCC Chairman, Tom Wheeler).

The Singapore government plays an important role in the economy, doing strategic economic and educational planning and they have invested in the oil, shipping, finance, media, Internet and biotech industries since World War II. I am not advocating a Singapore model for the US, but neither should we ignore possible steps local and national government can take to increase competition in the Internet service market.

The Straits Times benefits from the strong Singapore Internet, but I suspect the government also offers direct or indirect subsidy. I understand that we don't want the government to control our press, although there is considerable precedent for US government support of broadcast and print media. That being said, the current US administration will doubtless do its best to eliminate what little federal support remains.

But, since Republicans favor free markets and decentralized choice when it comes to health care, energy and schools, why not the press? How about media vouchers for voting age adults? Individuals would be free to allocate their media subsidy as they see fit -- to the New York Times or Breitbart, NPR or Rush Limbaugh. Milton Friedman might have even gone for that.

Saturday, February 25, 2017

Two approaches to routers in space -- SpaceX and OneWeb

Competing global ISPs would be of great value to mankind.

OneWeb collaborators and investors (Source)
Two companies hope to revolutionize the Internet by providing global connectivity using constellations of low-earth orbit satellites -- Elon Musk's SpaceX and Greg Wyler's OneWeb. It seems that SpaceX gets a lot more publicity than OneWeb, but both are formidable.

They have the same goal, but their organizations are dissimilar. SpaceX is integrated -- building the rockets, satellites and ground stations themselves -- while OneWeb has a number of collaborators and investors, including Bharti Enterprises, Coca-Cola, Intelsat, Hughes, Totalplay Telecommunications, Virgin Galactic and Softbank.

One strategic investor, Softbank, invested $1.2 billion last December and was given a board seat. OneWeb says they have now raised enough capital to finance the remainder of the project with loans.

OneWeb had planned to build 900 satellites and initially launch 648, but Wyler says Softbank has encouraged them to be more aggressive and he is considering adding an additional 1,972 satellites. Doing so would dramatically increase the total capacity of the system. Regardless, their goal is to connect every school by 2022 and "fully bridge the digital divide" by 2027.

Teledesic animation
Critics of the SpaceX and OneWeb projects argue that they will not be able to compete with terrestrial wireless and they also run the risk of causing "space junk" collisions in low-earth orbit. Others counter that it will be decades before ubiquitous, high-speed wireless connectivity reaches the majority of the people on Earth and the odds of such collisions are very small at such high altitudes.

(Teledesic, a similar project, failed in the 1990s, but launch and communication technology have improved dramatically since that time and Internet connectivity has become much more valuable).

What if one of these companies succeeds and the other fails? That would leave the winner with a monopoly in much of the rural and developing world. It is even conceivable that they could compete effectively with terrestrial ISPs -- in access or backbone networks. Would global ISPs require unique regulation and, if so, what should it be and who has the power to do it?

Los Angles - Punta Arenas 5 satellite hops, 14 terrestrial hops

I'm not smart enough to answer the critics who raise difficult questions, but I hope SpaceX and OneWeb both succeed -- competing global ISPs would be of great value to mankind.

(For more background and news on this topic, click here).

Update 3/1/2017

The satellite Internet strategies of OneWeb and SpaceX have diverged further with a proposed merger between OneWeb and Intelsat. (Softbank is an investor in both companies).

If the merger is completed, they will integrate their geostationary (Intelsat) and low-earth orbit (OneWeb) networks, enabling them to have global coverage quickly with mixed high and low-latency service, depending on the customer's location and requirements.
Presumably many current Intelsat broadband customers would transition to the OneWeb network as it becomes available and OneWeb customers will be able to offer mixed-speed service globally. As shown here, Intelsat would also bring regulatory approval access to 200 countries and territories to the combined company, but I wonder if those agreements would have to be renegotiated.

If the merger is approved, they will face a stiff challenge in integrating both the communication technology and the marketing/business models, but this is an interesting twist.

Monday, January 30, 2017

Do-it-yourself rural fiber

M-PAC cable
I doubt that any elementary school in the US has fiber to the premises, but, in 2013, an elementary school in rural Bhutan was connected to the Internet using optical fiber in the "last mile."

They were able to connect the school because the cabling they used, metal-packed armored cable (M-PAC), which is modeled on undersea cables, does not have to be in a protective duct. It is 4mm in diameter, light and flexible, so it can be installed by supervised volunteers or unskilled workers.

As shown below, a portion of the cable to the school is buried in a hand-dug ditch and another link is suspended overhead:

The cable used in this installation was supplied by OCC Corporation, but last June the International Telecommunication Union (ITU) adodpted a standard for "low-cost sustainable telecommunications infrastructure for rural communications in developing countries," L.1700.

As a framework standard, L.1700 is largely technology-neutral. Technology-specific best practices are provided by supplement texts such as ITU-T L Supplement 22, which specifies the design of a low-cost, terabit-capable optical cable that can be deployed on the ground’s surface with minimal expense and environmental impact. For more on the standard and it's intended application, check this post.

We have major fiber backbones in large cities -- might we also have do-it-yourself backbones in rural villages?

Thursday, January 05, 2017

History is written and revised by the winners -- can the Internet Archive change that?

Kremvax during the Soviet coup attempt
I was naively optimistic in the early days of the Internet, assuming that it would enhance democracy while providing "big data" for historians. My first taste of that came during the Soviet coup attempt of 1991 when I worked with colleagues to create an archive of the network traffic in, out and within the Soviet Union. That traffic flowed through a computer called "Kremvax," operated by RELCOM, a Russian software company.

The content of that archive was not generated by the government or the establishment media -- it was citizen journalism, the collective work of independent observers and participants stored on a server at a university. What could go wrong with that?

Mumbai terrorist attack
The advent of the Web and Wikipedia fed my optimism. For example, when terrorists attacked various locations in Mumbai, India in 2008, citizen journalists inside and outside the hotels that were under attack began posting accounts. The Wikipedia topic began with two sentences:
The 28 November 2008 Mumbai terrorist attacks were a series of attacks by terrorists in Mumbai, India. 25 are injured and 2 killed.
In less than 22 hours, 242 people had edited the page 942 times expanding it to 4,780 words organized into six major headings with five subheadings. (Today it is over 130,000 bytes, revisions continue and it is still viewed over 2,000 times per month). What could go wrong with that?

The Arab Spring
The 2011 Arab Spring was also seen as a demonstration of the power of the Internet as a democratic tool and repository of history. What could go wrong with that?

What went wrong

The problem is that the Internet turned out to be a tool of governments and terrorists as well as citizens. Furthermore, historical archives can disappear or, worse yet, be changed to reflect the view of the "winner."

Our Soviet Coup archive was set up on a server at the State University of New York, Oswego, by professor Dave Bozack. What will happen to it when he retires?

If someone tried to delete or significantly alter the Wikipedia page on the Mumbai attack, they might be thwarted by one of the volunteers who has signed up to be "page watchers" -- people who are notified whenever the page they are watching is edited. We saw a reassuring demonstration of the rapid correction of vandalism in a podcast by Jon Udell. That was cool, but does it scale? Volunteers burn out. The page on the Mumbai attacks has 358 page watchers, but only 32 have visited the page after recent edits.

Even if a Wikipedia page remains intact, links to references and supporting material will eventually break -- "link rot." If our Soviet Coup archive disappears after Dave's retirement, all the links to it will break.

By the time of the Arab Spring, we were well aware of our earlier naivete -- the Internet was already being used for terrorism and government cyberwar and the dream of providing raw data for future historians and political scientists was fading.

The Internet Archive

Soviet coup archive from Internet Archive
I was slow to understand the fragility of the Internet, but others saw it early -- most importantly, Brewster Kahle, who, in 1996, established the Internet Archive to cache Web pages and preserve them against deletion or modification. They have been at it for 20 years now and have a massive online repository of books, music, software, educational material, and, of course, Web sites, including our Soviet Coup archive. As shown here, it has been archived 50 times since October 3, 2002 and it will be online long after Dave retires -- as long as the Internet Archive is online.

Khale understands that saving static Web sites like the Soviet Coup archive only captures part of what is happening online today. Since the late 1990s, we have been able to add programs to Web sites, turning them into interactive services. As such, he has recently begun archiving virtual machine versions of interactive government services and databases.

Khale is understandably concerned by the election of Donald Trump, who has demonstrated a keen ability to exploit the Internet and a disregard for truth. As such, he is raising money to create a backup copy of the Interent Archive in Canada and working to archive US Government Web sites and services.

The Internet is inconceivably large and growing exponentially. There is no way the Internet Archive can capture all of it, but it is the leading Internet-preservation organization today. Khale and his staff will continue their work and will inspire and collaborate with other relatively specialized efforts like that of climate scientists who are working to preserve government climate-science research results, data and services.

For more on the Internet Archive check out the following PBS News Hour segment (9m 12s):

You can read the transcript here.

I'd also recommend listening to this short (5m 14s) podcast interview of Brewster Kahle. He describes the End of Term project -- a collaborative effort to record US government (.gov and .mil) Web sites and services when a new administration takes over. He describes deletions and modifications from 2008 and 2012 and feels a special urgency today for obvious reasons.

You can read a transcript of the interview here.

Update 1/6/2017

The Internet Archive has launched the Trump Archive with 700+ televised speeches, interviews, debates, and other news broadcasts. Mention by a fact-checking site was the "signal" used for inclusion of a video and links to the fact-check document are included in a companion spreadsheet. I hope they use speech recognition to produce searchable transcripts as well.

Too bad we did not have Trump and Clinton archives during the campaign -- I hope we will have similar, timely archives in the future. One can even imagine similar archives for state and local campaigns if a crowd-sourcing system were developed.

Update 1/7/2017

There is an annotated PowerPoint presentation on citizen journalism here. I use it in teaching an Internet literacy class and there is a note on my PowerPoint presentation style here.

Tuesday, January 03, 2017

Package delivery -- the other "last mile" problem

We've had bad luck with package delivery during the last six months:
  • An order of kids rain boots from Walmart was stolen from our front porch. Unfortunately, a small box containing a ring from TheRealReal was delivered at the same time and was also stolen.
  • Walmart replaced the rain boots and TheRealReal gave us a refund, but my wife was disappointed not to get the ring.
  • We received a package from TheRealReal via Federal express. It should have contained a bracelet, but it was empty. Again, we received a refund, but not the gift. It may have been taken by someone at TheRealReal or Federal Express.
  • We ordered a pressure cooker from Amazon. The package it came in was marked "fragile" but was in poor condition. We opened it, saw two dents in the pressure cooker and returned it for a refund.
  • We ordered a blanket from SweetDreamsHome, an Amazon Marketplace retailer. The order was placed on December 14 and scheduled for delivery. We planned to be out of town on the scheduled delivery date, so requested a different and were assured it would arrive on December 23. It did not arrive on that date, so we contacted Amazon. We were assured that it would arrive on December 28th. It did not. When it did not arrive on the 29th, we cancelled the order. It arrived on the 30th.
Amazon and the others were extremely polite and responsive and we received prompt, no-hassle refunds, but we were disappointed, a Christmas gift was late and we had to be worrying that a package might come while we were out and unable to sign for it or, worse, that it would be stolen.
I checked the American Customer Satisfaction Index of the Consumer Shipping and Internet Retail industries and found that their scores of 80 out of 100 put them in the top six of 43 industries surveyed. (Internet service providers were ranked last because there is little competition in the industry).

That being said, the survey only considers the US Postal Service (74), UPS (80) and Federal Express (82). The private companies are rated higher than the Postal Service and all three have been relatively stable over time. (The US Postal Service moved up in the late 1990s, while UPS and Federal Express have slipped a little).

Fortune magazine says Silicon Valley venture capitalists are giving up on on-demand delivery and I am not expecting on-demand drones or robots or self-driving delivery trucks any time soon. (If they do, the thieves may start stealing drones and robots as well as packages). Are vendor-agnostic local pickup locations a solution?

Maybe this was a run of bad luck and we plan to keep shopping online, but not as frequently.

Sunday, December 18, 2016

The International Consortium of Investigative Journalists discovers Tillerson ties to offshore company used in Russia deal

The Panama papers reveal Secretary of State nominee Rex Tillerson's ties to Russia and offshore companies -- the first of many such revelations?

The Panama Papers is a collection of 11.5 million documents (2.6 terabytes) that was leaked by an anonymous source to Süddeutsche Zeitung (SZ), a German newspaper. The documents were from the internal files of Mossack Fonseca, a Panamanian law firm that creates anonymous offshore companies around the world. The database on 320,000 offshore companies may be accessed here.

SZ did not have the staff and resources to analyze that many documents, so they decided to cooperate with the International Consortium of Investigative Journalists, a global network of more than 190 journalists in more than 65 countries who collaborate on in-depth investigative stories.

(The story of this massive, Internet-based collaboration is amazing in its own right. For more on the ICIJ and the methodology of this investigation, check out this excellent 15 minute podcast, with transcript).

The ICIJ has now turned it's attention to the Trump administration and has discovered that Rex Tillerson, ExxonMobil CEO and Secretary of State nominee was a director of an offshore company in the Bahamas that is at the heart of Exxon’s close business dealings with Russia.

The ICIJ reports that:
The records show Tillerson’s direct involvement in Exxon’s extensive network of companies based in the Bahamas. ExxonMobil created at least 67 companies based in the island tax haven, which were involved in operations spanning from Russia to Venezuela to Azerbaijan, according to ICIJ’s documents from the Bahamas corporate registry.
An ExxonMobil spokesman said that it incorporates in the Bahamas because of the “simplicity and predictability” of the country’s laws for setting up companies and that "Incorporation of a company in the Bahamas does not decrease ExxonMobil’s tax liability in the country where the entity generates its income.”

This may be legal and may not be depriving the US of tax revenue, but it does raise questions of Russian influence and conflict of interest. Tillerson currently holds an estimated $228 million in Exxon stock, whose value stands to be affected by State Department policies on issues from climate change to sanctions against Russia.


The ICIJ promises to continue investigating the Trump administration -- stay tuned.
Update 12/19/2016

This is not the only result of the ICIJ investigation of the Panama Papers. For example, an earlier investigation revealed that Mossack Fonseca had been used to "create a string of companies in offshore financial havens that allowed it to sidestep the U.S. embargo in its commercial operations." They have identified at least 25 companies registered in the British Virgin Islands, Panama and the Bahamas that are linked to Cuba, enabling the Cuban government to import and export goods and invest funds abroad. Another investigation led to the resignation of the Prime Minister of Iceland.

The ICIJ promises to continue investigating the Trump administration -- stay tuned.

Thursday, December 15, 2016

Why we need the Washington Post, New York Times et al

We can easily afford to lose publishers like Gawker Media, but not papers like the Washington Post and the "failing" New York Times.

Donald Trump's choices to head the Energy and Interior departments and the Environmental Protection Agency are climate-change "skeptics" and they support and are supported by the oil industry. This has led some climate scientists to initiate projects to back up climate-sicence research and data.

The Washington Post published a well researched article on the concern of the climate scientists with links to many supporting articles. Donald Trump routinely denigrates the "mainstream media," but this article is a terrific example of what the press can and must do.

The Internet has disrupted the business model of newspapers like the Washington Post and New York Times and the Trump administration poses another threat.

Peter Theil is a member of the Trump transition team and a very rich Silicon Valley investor. Gawker Media alienated him by publishing the fact that he was gayand Theil retaliated by secretly financing a law suit for Hulk Hogan who had also been embarrassed by Gawker. The suit bankrupted Gawker Media.

Donald Trump frequently threatens to sue adversaries. Can we imagine him or a supporter like Theil suing the Washington Post?

Maybe, maybe not, but he will surely continue using his "bully pulpit" for ad hominem attacks against publications that fact-check and criticize him. For example, consider these tweets about the "failing" New York Times:

Trump nicknames -- lying, little ... now failing Source

In 2013, Jeff Bezos, founder of Amazon.com, purchasted the Washington Post from the Graham family for $250 million -- a lot of money for you and me, but not much for Donald Trump or Peter Theil.

Trump has has threatened Bezos, saying he has "a huge antitrust problem because he's controlling so much, Amazon is controlling so much of what they are doing." He added that Bezos is "using The Washington Post, which is peanuts, he's using that for political purposes to save Amazon in terms of taxes and in terms of antitrust."

The Internet has cost newspapers dearly and the Graham family might have been vulnerable to an attack by Trump if they had not sold it. At the time Bezos bought the Washington Post, there was a lot of speculation as to his motivation. I don't know why he bought it, but I am glad he did, because he can afford to defend it.

We can easily afford to lose publishers like Gawker Media, but not papers like the Washington Post and the "failing" New York Times.

Backing up climate-science data

It is nearly inconceivable that Trump would order the deletion of climate-science data -- a modern book burning -- but one can imagine large budget cuts for climate-science research, making it impossible to maintain and update this sort of public data.

Climate scientists have kicked off at least two projects to create backup copies of their research and data.

One is Climate Mirror, which is part of an ad-hoc project to mirror public climate datasets before the Trump Administration takes office -- to make sure these datasets remain freely and broadly accessible.

Another is a hackathon that will be hosted on December 17th at the University of Toronto in collaboration with the Internet Archive End of Term project, which seeks to archive the federal online pages and data that are in danger of disappearing during the Trump administration. (Note that they have done the same for earlier administrations).

For example, NASA recently released data showing how temperature and rainfall patterns worldwide may change through the year 2100 because of growing concentrations of greenhouse gases in Earth’s atmosphere.

The post announcing the dataset states:
The dataset, which is available to the public, shows projected changes worldwide on a regional level in response to different scenarios of increasing carbon dioxide simulated by 21 climate models. The high-resolution data, which can be viewed on a daily timescale at the scale of individual cities and towns, will help scientists and planners conduct climate risk assessments to better understand local and global effects of hazards, such as severe drought, floods, heat waves and losses in agriculture productivity.

"NASA is in the business of taking what we’ve learned about our planet from space and creating new products that help us all safeguard our future,” said Ellen Stofan, NASA chief scientist. “With this new global dataset, people around the world have a valuable new tool to use in planning how to cope with a warming planet.
The climate-science community is obviously alarmed by Donald Trump's appointments of Ryan Zinke, who characterizes climate change as “unsettled science," as Secretary of the Interior, Rick Perry, who once could not recall the name of the department, but remembered that he did want to eliminate it, as Secretary of Energy and Scott Pruitt, who consistently opposes regulation, to head the Environmental Protection Agency.

These men are all supporters of and supported by the oil industry.

The Trump transition team also requested a list of the names of Energy Department people (contractors and employees) who have worked on climate science and the professional society memberships of lab workers.

It is nearly inconceivable that Trump would order the deletion of climate-science data -- a modern book burning -- but one can imagine large budget cuts for climate-science research, making it impossible to maintain and update this sort of public data.

Update 12/29/2016

Check out this excellent, short (5:14) interview of Internet Archive founder Brewster Kahle. The interview begins with climate scientist Eric Holthaus talking about the effort to archive climate research, then Khale goes on to say more about how and why they archive government (.gov and .mil) Web sites when a new administration takes over.

He said 83% of the .pdf files on government sites were deleted between 2008 and 2012. In addition to Web pages, they will be archiving virtual machine versions of interactive government services and databases. (As noted above, those are vulnerable to defunding).

When asked for an example of the value of the archive, Khale mentioned the press release announcing George Bush's ironically famous "Mission Accomplished" speech on the deck of an aircraft carrier. As shown below, the headline reads "President bush announces combat operations in Iraq have ended" and the first sentence qualifies the headline by saying "major" combat operations have ended. Khale said that a couple of weeks later "major" was added to the title and a couple months later, the press release was deleted.

Excerpt from press release on "Mission Accomplished" speech

The Internet is potential providing raw data for historians -- it should be complete and accurate.

If you would like to see a video of the entire speech, the Internet Archive has preserved that as well:

Update 12/30/2016

The following is a transcript of Bob Garfield, co-host of the podcast On The Media, interviewing Brewster Khale, founder of the Internet archive and a partner in the End-of-term Project with a lead-in question for on climate-science research Eric Holthaus of Slate Magazine.

Bob: Meanwhile a small army of volunteer archivists, scientists and advocates have been working to save the government climate change research that already exists

Eric: at NASA and NOA that takes the temperature of the planet from weather stations from satellites from ocean buoys.

Bob: Meteorologist Eric Holthaus spoke to NPR about his effort to save government climate data.

Eric: Sometimes these data sets are only stored in United States government servers so there hasn't really been an effort to catalog those in other countries because we haven't thought it was necessary before

Bob: The Internet Archive on the other hand has given a lot of thought to what gets lost in presidential transitions. Every week the archive tapes three hundred million Web pages and every four years it enlists a bunch of volunteers to make copies of government Web sites as a hedge against what the next administration may choose to delete. It's called The End-of-term web archive and for some reason this year the organizers are getting a lot more offers of help. Brewster Kahle founder of the Internet Archive says that this year his team also is backing up its data to Canada

Brewster: When the election went the way that it did, it was a bit of a surprise, so we looked through the television archive at what President-elect Trump said about freedom of the press and about the Internet and what we found was shocking. He wanted to close up parts of the Internet that there was mocking of freedom of the press. This was kind of a wake-up call and we said let's make sure we have a copy in some other location.

Bob: What are your priorities? How does it work?

Brewster: So the Internet Archive works with the Library of Congress, University of North Texas -- now a growing list of groups to try to do as best we can to record the information that's available on the Web sites and now the web services that have been made available on .gov and .mil Web sites. We found in 2008, 83 percent of the PDFs that were available back then are no longer available even by 2012. So with an 83 percent loss rate when the Obama administration came on board we're likely to see something like it maybe even more with the Trump administration.

So we're coordinating activities to go and archive web pages and we're reaching out to federal webmasters to go and see if we can keep whole services up and running. Can we take virtual machine versions of the databases that they're running and be able to run them in snapshot form so that we can keep these services going as they were in 2016 in the future?

Bob:Give me some examples of when the federal web archive has come in handy. Was there something that you and disappeared that you were super glad to have archived?

Brewster: Oh the anecdotes go on and on. Example -- there is a press release from the White House during the George W. Bush administration when he stood on an aircraft carrier and declared “mission accomplished.” And the headline of that press release was combat operations in Iraq had ceased but a couple of weeks later they changed the headline and said major combat operations had ceased with no notice that it had changed. The only reason why we know is because we had archived both versions. And then a couple of months later the press release went away completely from the web. You know what is more Orwellian is it changing a press release that's in the past or is it disappearing completely?

Bob: What are you most worried is going to disappear in a Trump administration?

Brewster: Frankly we have no idea. This upcoming administration is very aware of the power of the Internet and how it can be manipulated -- how you can go and push things out in the middle of the night and use the journalist system in ways that are really pretty blatant. So let's at least keep a record of it.

Bob: We have just experienced the interference in a political campaign by outsiders. Is this archive secure -- I mean really secure against hacking, against intrusion?

Brewster: The history of libraries is a history of loss. Libraries are burned. That's what happened in the Library of Alexandria. It'll be what happens to us -- just don't know when. So let's design for it. Let's go and make copies in other places. Let's make sure people want universal access to all knowledge, that they want education based on facts. Let's go and make sure that there is an environment that supports libraries. That's the only way that in the long term we're going to survive. And the copies that are maybe now unique at the Internet Archive will survive based on all sorts of changes whether it's earthquakes or institutional failure or law changes.

Bob: Brewster as always many thanks.

Brewster: Thank you very much.

Bob: Mr. Khale is the founder of the Internet Archive and a partner on the End-of-term Project.

Khale's interview was part of longer podcast episode called Hurry Up. They discussed other steps President Obama could take during the last weeks of his term. The suggestions included disclosing information on contributions by government contractors, surveillance and the drone program, closing Guantanamo and clemency. The episode ends with a discussion of the nature of time by science writer James Gleick.

Finally, I created the interview transcript using a nifty service called PopUpArchive. You simply upload a sound file and wait for the text version to be posted ready for download. It takes a little proofreading and editing, but it is a lot faster than manual transcription and as this Microsoft Research report shows, we can look forward to more accurate speech recognition in the future.

Thursday, December 08, 2016

Trump's China tweets -- data for historians, political scientists, journalists and us

Trump's tweets and other posts provide us with an unprecedented stream of current information and data for political scientists, journalists and future historians.

The New York Times has published a thorough analysis of Donald Trump's recent phone conversation with the president of Taiwan. It takes a multifaceted look at the call, asking whether it was a "diplomatic gaffe or a calculated new start" in our relationship with China.

Only Trump, his advisors and perhaps some of the people he has been interviewing for Secretary of State can answer that question, but we can get clues as to Trump's thinking by looking at his Twitter stream.

A search of his Twitter stream for the word Taiwan, returned only four tweets:

The two October tweets are anti-Obama campaign statements.

The tweet announcing the call says "CALLED ME" in caps. Was that Trump crowing about his stature or was it intentionally saying he had not initiated the call? I cannot know, but I am certain that this was not a casual call -- it was planned and scheduled by both sides in advance.

The latest tweet justifies the call and serves as a message to China and Trump's constituency. (I have to reluctantly admit that I agree -- pretending that Taiwan does not exist is absurd).

Trump's tweets do not provide definitive answers, but they do give us more information about what is going on than we are used to.

"China" tweets

Since Trump is focusing on China, I searched of his Twitter stream for the word China. Twitter returned 276 tweets -- here are the earliest four:

Trump has been posting anti-China tweets for nearly six years. The first had little engagement -- one reply, 73 retweets and 26 likes -- while the latest one has had 22k replies, 39k retweets and 122k likes so far. He was already campaigning at the time of the first tweet, which refers to a site called shouldtrumprun.com. (Today that site contains only a copy of a statement by the Federal Election Commission saying he was eligible to run).

Who is the intended audience for these tweets? No doubt, the early tweets were intended for Trump supporters -- Breitbart readers and Limbaugh listeners -- but future tweets might also be for the general public and the Chinese government.

I have no doubt that both our State Department and the Chinese Foreign Service are well aware of the issues on which our nations co-depend and where we have conflicts, but discussions of such things are traditionally held in private. Whatever you think of Trump, he is providing us with a degree of transparency we are not used to in our politicians and civil servants.

Listening to a fireside chat
New media are mastered by new politicians and Trump's use of social media is reminiscent of the fireside chats President Roosevelt used to communicate with the American people and others when radio became ubiquitous.

If I were a political scientist, I would begin looking at these and Trump's other tweets and posts as research data, ripe for content analysis and fact-checking. They will also be data for historians one day. (The archive of network traffic during the 1991 Soviet coup attempt might be the earliest example of historical data online).

One thing is for sure -- I hope he keeps tweeting after becoming president.

Tuesday, December 06, 2016

I hope Trump keeps tweeting

I hope Trump keeps up his tweeting. They say our eyes are windows to our souls, his late-night tweets are a window to his.

Since I have a blog on the Internet in Cuba, I took a look at Trump's tweets, hoping to learn something about his likely Cuba policy. I searched his Twitter stream for tweets with the words Cuba or Cuban and Twitter returned 27 results, but only three were what I was looking for.

I was surprised to see that 20 of the tweets were about Mark Cuban, an entrepreneur, business man and outspoken Trump critic and four related to President Obama. What do those tweets reveal about Trump?

Two of the tweets illustrate his competitive nature.

In this tweet, he brags (with reason) that his reality TV show, The Apprentice, was a bigger hit than one Cuban was on, The Shark Tank.

(NBC later severed relations with Trump because of his remarks during the campaign).

This tweet refers to the Dallas Mavericks, a professional basketball team owned by Cuban:

The next tweet illustrates Trump's proclivity for personal, ad hominem attacks:

(Trump's physique has also been ridiculed).

Three of trump's Cuba tweets were shots taken at President Obama during the campaign, for example these:

An earlier tweet about President Obama was as goofy as Trump's "birther" campaign:

It turned out that only three of the tweets pertained to my initial question:

They give us a clue as to his posture on Cuba during and after the campaign, but I suspect his hard line will be tempered by practical economic and political factors. Regardless, Trump's tweets reveal more about him than about his Cuba policy.

I plan to repeat my Twitter search "Cuba from:realdonaldtrump" from time to time to see how his views evolve. I hope Trump keeps up his tweeting. They say our eyes are windows to our souls, his late-night tweets are a window to his.

Friday, November 25, 2016

Is the Internet becoming a vast wasteland?

I've written posts about trolls in Cuba, where Operation Truth is said to use a thousand university-student trolls and trolls in China where government workers fabricate an estimated 488 million social media posts annually.

Now we are reading about Russian government trolls. Just before the election, this post documented Russian trolling and warned that "Trump isn’t the end of Russia’s information war against America. They are just getting started."

After the election a new site, PropOrNot.com (propaganda or not) came online. Their mission is outing Russian propaganda using a combination of forensic online sleuthing and crowdsource reporting and they have compiled a list of 200 sites that rapidly spread stories written by Russian trolls. (More about PropOrNot here).

But, is PropOrNot what it claims to be? The people behind the site remain anonymous (for understandable reasons) and their domain name registration is private. How do they determine that a site is home for Russian content? Is there a chance that they are pro-Clinton, sour-grapes trolls? Might trolls and hackers figure out ways to game ProOrNot and get sites they oppose blacklisted?

Hmmm -- I wonder if the US government hires trolls and, if not, should they? How about Canada? Chile? Zambia? How about Exxon Mobile trolls or McDonalds trolls? Is it trolls all the way down?

The fake news and trolling revealed during the last few months of the US political campaign has sowed doubts about everything we see and read online. We're beginning the transition from "critical thinking" to "paranoid thinking."

Newton Minow
In 1961, Newton N. Minow gave a talk to the National Association of Broadcasters in which he worried that television was becoming a "vast wasteland:"
But when television is bad, nothing is worse. I invite each of you to sit down in front of your television set when your station goes on the air and stay there, for a day, without a book, without a magazine, without a newspaper, without a profit and loss sheet or a rating book to distract you. Keep your eyes glued to that set until the station signs off. I can assure you that what you will observe is a vast wasteland.
Will the Internet become a vast wasteland? Newton Minnow was correct, but there were and still are oases in the television wasteland. In spite of the trolls, fake news sites, troll-bots, etc. the Internet is and will remain replete with oases, but we cannot ignore the wasteland.

Update 11/28/2016

I reached out to PropOrNot, pointing out that they do not identify themselves and their domain registration is private and asking how I could know they were not posting false claims themselves. They replied that "We sometimes provide much more background information about ourselves to professional journalists."

They have now posted a document on their methodology, showing how they select sites for their list. They are not saying the sites are paid trolls, but that they publish information that originates on Russian government sites -- that they disseminate Russian propaganda.

At least one of the sites on their list, The Corbett Report, has refuted the claim that they are pro-Russian, but they do not address the question of their distributing material that originated on Russian sites.

Monday, November 21, 2016

Teaching slides on the political impact of the Internet

I teach a class on the applications, technology and implications of the Internet and we begin each week with a discussion of current events relevant to the class. This semester many of those discussions have included material on the implications of the Internet for politics.

I've gone through my discussion slides for this term and put those that deal with the political impact of the Internet in a single, annotated PowerPoint file.

The slides focus on the current election, but also establish context by covering the use of then-new media in earlier elections, starting with radio, and other disappointing political uses of the Internet.

The slides are in chronological order as we have gone through the semester. If you are teaching a related class -- perhaps in political science -- you might find something useful. (I will continue adding new material -- suggestions welcome).

Update 12/1/2016

I've added new slides dealing with the outing of fake news purveyors, truth versus freedom of speech, risk-limiting audits, a Stanford study of junior high, high school and college students reading of Internet news and the possibility that the Internet might become a "vast wasteland."

Update 12/14/2016

I've added new slides dealing with National Security Advisor nominee General Mike Flynn's tweets concerning fake news. He later deleted one of the embarrassing tweets, but it had already been cached by the Internet archive.

Update 12/17/2016

I've added slides dealing with General Flynn, European fake news, Tiananmen Square, the consequences of fake news, good journalism, Trump's continuing post-election lies and Facebook's effort to identify fake news.

Update 12/20/2016

I've added slides on "Alt Right" gaming of Google search and Google's effort to thwart them.

Update 12/21/2016

I've added new slides on "Alt Right" gaming of Google (and Bing) search, Google's effort to thwart them and a Pew Research survey on the public's view of fake news.

Update 12/23/2016

I've added new slides on fantasy versus reality (examples from Trump and Bill Clinton) and whether they matter and Facebook's moves to combat fake news after first dismissing their role.

Update 12/27/2016

I've added new slides on the possibility of real-world consequences to fake news and Russian political hacking in Germany, Italy and the US.

Update 1/6/2017

I've added new slides about the Panama Papers and what they reveal about Secretary of State nominee Rex Tillerson. (Collaborative analysis of the Panama Papers by journalists is a positive application of the Internet in politics).

The new slides are located here.

The cumulative slide deck is located here.

Update 1/9/2017

I've added new slides tracing the impact of a single fake story on the Breitbart News Web site along with some background on Breitbart's Steve Bannon.

The new slides are located here.

The cumulative slide deck is located here.

Update 1/16/2017

I've added new slides dealing with Trump's writing style on Twitter and his reaction to his briefing on Russian hacking by intelligence agencies. (His response also provides an illustration of his Twitter writing style.

The new slides are located here.

The cumulative slide deck is located here.

Update 1/17/2017

I've added new slides dealing with Facebook's efforts to flag fake news in the US and Germany and possible fact-checking collaboration.

The new slides are located here.

The cumulative slide deck is located here.

Update 1/19/2017

I've added new slides dealing with the Internet Archive and others in archiving Internet content that might be deleted or altered for political purposes.

The new slides are located here.

The cumulative slide deck is located here.

Tuesday, November 15, 2016

A real-names domain-registration policy would discourage political lying.

I've discussed the role of the Internet in creating and propagating lies in a previous post, noting that Donald Trump lied more frequently than Hillary Clinton or Bernie Sanders during the campaign.

Now let's look at fake news like the claim that Pope Francis had endorsed Trump. The fake post features the following image and includes a "statement" by the Pope in which he explains his decision.

The post evidently origniated on the Web site of a fake news station, WTOE 5. Avarice, not politics, seems to be the motivation for the site since it is covered with ads and links to other “stories” that attack both Clinton and Trump.

WTOE 5 states that it is a satirical site on their about page, but how many readers see that? Other sites do not claim satire. For example, the Christian Times about page says nothing about satire, but does assert that they are not responsible for any action taken by a reader:
Christian Times Newspaper is your premier online source for news, commentary, opinion, and theories. Christian Times Newspaper does not take responsibility for any of our readers' actions that may result from reading our stories. We do our best to provide accurate, updated news and information
The Christian Times "editorial" policy is similar to that of WTOE 5 -- they published pre-election news stories on thousands of dead people voting in Florida, hacking of voter systems by the Clinton campaign, Black Panthers patrolling election sites, etc. As soon as the election was over, they informed us that Hillary Clinton had filed for divorce. Don't believe it? Here is their evidence:

Given the WTOE 5 claim to be satire or the Christian Times eschewing responsibilty for actions taken by readers, I suspect that unliess Pope Francis or Hillary Clinton sues, there is no legal recourse.

The dirty tricks during this election remind me of the Watergate burglary, but, unlike Watergate, it is not clear that a law has been broken. In the Watergate case, a crime was committed and the burglars were convicted and sent to prison in 1973. In 1974 investigators were able to establish a White House connection to the burglary and, under threat of impeachment, President Nixon resigned.

Would it be possible to establish a connection between a Web site like "WTOE 5 News" and the Trump campaign?

A Whois query shows us that the domain name Wto5news.com was registered by DomainsByProxy.com. We can see the address, contact information and names of people at DomainsByProxy.com, but the identity of the person or organization registering the domain name is private.

I also checked the Whois record for the Christian Times. It turns out that DomainsByProxy.com is also the registrar for Christiantimesnewspaper.com and the registration is also private.

I am not a lawyer, but I suspect that a request for a subpoena to get the contact information of a long list of people registering domain names for misleading Web sites would be seen as a "fishing expedition" by the courts.

I understand the wish to protect the privacy of a person or organization registering a domain name, but there is also a public interest in discouraging sites like Wto5news.com. A verifiable, real-names policy for domain registration would discourage this sort of thing. The WELL, an early community bulletin board system, adopted such a policy years ago. Their slogan is "own your own words" and it serves to keep discussion civil, stop bullying and lying, etc.

Trump supporters seem to worry a lot about voter fraud. They advocate easing mechanisms for challenging a voter's registration and encourage strict requirements for proof of identity and residence. There is more evidence of demonstrably fraudulent political information on the Internet than fraudulent voting. If their concern is genuine, they should support a real-names policy for domain registration.

If warrants will not pass legal muster and a real-names policy is unrealistic, someone might be tempted to follow the example of the Trump supporters who hacked the Democratic National Committee and resort to hacking registrars to get contact information of their private clients. Maybe Julian Assange could distribute what they find on WikiLeaks.

Update 11/17/2016

I received some comments on this post from an attorney.

For a start, he said labeling something as "satire" was irrelevant because the defense would be 1st amendment free speech. He said there might be a slight chance for the "shouting fire in a crowded theater" argument, but he and this Atlantic Monthly article agree that that is a long shot. He also said there might be a remote chance of a false advertising claim succeeding, especially if it were against a person working on the Turmp campaign like Steve Bannon of Breitbart or Sean Hannity with his popular radio and TV shows. Regardless, it would be necessary to show that their behavior had altered the election result (for president or "down ballot" contests).

I agree that it would be nearly impossible to show that a single site or lie had provided the margin for Trump's victory, but I do believe that rigorous survey research by a reputable organization could demonstrate that the marginal impact of fake sites and posts in the aggregate was sufficient to elect Trump and I hope that such research is conducted.

Regardless, a true-names policy would help investigators looking for possible connections between the Trump campaign and intentional, systematic Internet misinformation. The revelation of the White House role in Nixon's "dirty tricks' was what mattered, not the conviction of the burglars.

Update 11/18/2016

The attorney who commented on this post (above) suggested that it would be relevant if Breitbart published a lying article.

After Khizr Khan spoke at the Democratic convention, Breitbart published a story saying he had deep ties to the government of Saudi Arabia, international Islamist investors, controversial immigration programs that wealthy foreigners can use to essentially buy their way into the United States and the “Clinton Cash” narrative through the Clinton Foundation.

That is a lot of deep ties, but it turns out the post was false.

The Breitbart story has had 167,000 Facebook engagements and the refutation has been shared 32,600 times on Facebook, Twitter, Google Plus, Reddit, by email and shared links combined.