MPEG - a Blight Against Video Quality

Watching music videos tonight, I realized something: MPEG video encoding is a crime against quality.

Back when bandwidths were small and expensive, MPEG made near-full-speed video possible. Without severe compression, we wouldn't of had digital cable until years later. But this amazing technology comes with a serious cost.

MPEG compression removes most of the data, and sacrifices some of the video quality to do it. The higher the compression rate, the less data you end up with (less to transmit or store), but the worse the quality gets. To my knowledge, there is no MPEG compression that doesn't lose at least some of the quality. "Some" quality loss that your eye doesn't really notice, is fine. But MPEG has some serious drawbacks, it screws up when the entire screen is changing a whole lot, quickly - like during explosions in an action movie.

I'm not going to explain about Key frames and all that - there's plenty of web sites where you can learn how MPEG transmits the full frame of video only once in a while, sending "what has changed on the screen since the last frame" most of the time.

I'm just going to say that MPEG is so wrong for many reasons.

In the old days, with analog TV, you could flip through channels very quickly. A high-quality VCR could "tune" from one channel to the next in about 1/10th of a second. This was awesome - you could flip through channels and tell very quickly whether you wanted to watch a channel or not. Experienced TV viewers could tell in an instant what was an ad and what was a show; it's a finely-honed skill. But with MPEG compression, this is no longer possible! MPEG only transmits a key frame about once a second, or less. So, when you tune into a new channel, you've jumped right in the middle of "this changed, that changed, and on the next frame, this changed, that changed"; your tuner has no idea what it's talking about until the next Key frame goes by! Visually, you see: nothing. Black screen. For up to a second, sometimes longer. Then the picture appears. It's really irritating!

What I'm saying is, no matter how fast your computer/HDTV/Bluray/Tivo/Roku/Google TV/Digital TV Tuner/cell phone gets, it will NEVER be faster at tuning. Never. Not with MPEG, anyway. It's not a "hardware is slow" issue - it is a design flaw in the MPEG compression itself.

Suppose you turned on CNN Headline News, and the title along the bottom says "Dow Jones average", and the news anchor person is saying "it went up 5 points just now... now it went down 3 points... now it went up 13 points... now it went down 6 points..." And you wonder, what is the Dow right now? Nothing on the screen is telling the exact number. You're getting second-by-second updates about the number, but you don't know what the number actually is. You don't know, because you didn't hear what the price was when they started reciting the changes in price; plus, you missed some of the changes in price they said earlier. Now imagine that once every hour they'll actually tell you the exact price - "it's 11524 right now", then they continue reciting all of it's ups and downs. Since you heard the last real price, you have a chance of tracking the number during the next hour - by adding and subtracting the intermediate numbers they tell you, from that top-of-the-hour number of 11524.

That's sort of how MPEG works. If you miss the top-of-the-hour number, you gotta wait an hour to hear it again; you can't tell anything useful until then. For MPEG it's about once a second, which is a long time in video-land. It's irritating when you're tuning channels.

And nobody complains about it. They just live with it. This pisses me off! There are technological things that could be changed to fix this problem. One fix would be to change the MPEG compression to transmit Key frames more frequently. Yes, this would take up more bandwidth per channel - and it's WORTH IT for quality improvement - but if nobody complains, it won't be done. Why should media companies change their ways, if people put up with what they have today?

Another possible trick would be to use multiple tuners, 2 or 3. When you're on channel 15, say, it's already tuned in channel 16 and 17, and is waiting for the Key frame to go by. If you suddenly change to channel 16, the channel 16 tuner actually saw the keyframe go by at some point during the last second - so it can show you the video image INSTANTLY. The old channel 15 tuner now can become the channel 18 tuner (so your box is tuned to channels 16, 17, 18, and you are viewing 16). This solves the problem so long as you don't try to flip channels faster than 3 per second. Not great, but somewhat better than today. But what if the person changes channels down, instead of up? Maybe you need 1-2 extra tuners in that direction, too. 5+ tuners? That's a lot of extra hardware, and extra cost, to work around a software problem! And what if the person jumps to a new channel that's not in sequence? You didn't have that channel tuned, so we're back to the full second-plus delay again. Maybe you need a tuner on EVERY channel? That's not a practical solution for your hardware to do. Frustrating.

The Key-frames-not-sent-often-enough problem hurts you, the viewer, in other ways, too. The most action-packed scenes in a movie are often RUINED by MPEG encoding. When everything is flashing on the whole screen and changing quickly - explosion with parts flying everywhere, there's no way MPEG can handle it - it has to transmit Key frames for nearly every frame, which is too much data, so the number of frames per second drops dramatically - sometimes to 2 frames per second! This most important part of an action movie is now ruined, reduced to a slow-moving slideshow of fireballs - ruining the height of excitement, and reminding you of uncle Jethro's neverending slideshow of last year's family wiener roast. Just think. All the trouble and expense Hollywood put into making this one-shot-only scene, and they can't even record it on DVD properly for their customers to see. Incredible.

Sometimes the tree outside my house partially blocks the digital TV dish in my back yard, so some of the digital data gets lost. If the lost data happened to be some of the changing-frames, I only notice a little blurriness of the picture and it goes away quickly. But if a Key frame got damaged or lost, look out! The entire screen freaks out with wild colors and crazy blockiness, for at least 1-2 seconds of time! Lot's of green, usually, which is strange. Anyway, it looks horrible, like the person on the screen was stuck in mud, and it's sticking to them as they move around; then just as suddenly, it's all clear and working again. The 1-2 second weirdness was because the change-frames were describing changes, but my TV had the wrong Key frame data to compare it against. The screen cleared up when the following Key frame came through undamaged. I always know it's time to trim the tree when this happens.

The MPEG standard was designed and chosen back when dialup Internet was standard, and digital video was a tiny rectangle on a computer screen only. Today, all this has changed. If you "only" have 1Mbps Internet, that's considered slow. Half the web pages now have video advertisements playing alongside your normal content. And a wide variety streaming movies and clips play easily from a variety of sites like YouTube and Netflix. We have more bandwidth now. Our computers are 8X faster, now, and have Terabyte hard drives in them for local storage. I think we can handle updating the way we use MPEG to use a little more data, and improve quality dramatically at the same time.

Boy it's a good thing people don't complain enough, or somebody might actually have to fix this blight. Sometimes I wish we could return to the speed and simplicity of the "good old days" of analog television.

tl;dr: MPEG sux. TV channels should use more bandwidth to give us QUALITY video for a change. TV's moving to the Internet anyway; everything is. Just do it already.

Governmental Spending Transparency

I think its time the government opened up its books for the public to see, in a way that isn't completely confusing and complicated. Its time that people have the ability to see the cost-impact of those things they directly voted for, as well as for decisions being made by their elected officials. (For the purposes of this article, I'm going to refer to the United States of America where I live; other countries should be able to do something similar).

If done right, this will solve many problems:

• people feel like the government is wasting lots of money on unknown things; less hatred of government organizations and incorrect feelings of governmental incompetence.
• people have no idea the relative size of costs within the government: is $5 million spent on something a lot of money, or not much money at all? If it's a lot, I want to study it and stop them if it's not the best usage of that money. If it's nothing, I want to ignore it and find something bigger to focus on.
• more ordinary people can become involved in a rudamentary understanding of what our government is accomplishing.
• TV media could not create outright lies about what our government is doing, how our taxpayers money is being spent. They couldn't get away with it anymore.
• TV media would not be able to take biased sides like they do today, to deceive people in two extreme ways (liberal and conservative). Media bias would be eliminated by true data access - it will be clear to everyone where they're doctoring the truth; they will not continue their deceptive ways.
• people are asked to vote, today, on Propositions allocating large dollar amounts for various purposes. How can people vote yes or no on issues without a clear understanding of all the related spending today?
• the biggest chunks of money must to go the best, right and proper use, in our country. Start fixing the big things first, and work your way down to smaller things. How can we do this today, without a clear understanding of how much is spent where and when? With a proper database, peer pressure (by public viewing of expenditures in a clear form) can solve the problem.

Example: should I vote "yes" to spend money fixing a dam that broke on a river? I can't vote yes or no with a clear conscience until I know some other answers: What other dams are still generating electricity? What other non-dam electricity are we generating? Is it enough? What percentagedid we lose by this dam's outage? What is the effect of this loss? What is the cost of voting no on this issue within 1 year, 5 years, 10 years? How much is the demand for electricity increasing, such that maybe we need the dam next year, but we don't quite need it this year? Is the money to be spent fixing this dam going to be paid back within 1 year of dam operation? 5 years? 10 years? What other states in our country have had dams break recently, and what did they do, fix it (or not)? What was the consequence of their decision? Does the population believe that was the right decision, now that some time has passed?

Simply quoting the exact confusing wording of the proposed law, and listing biased-arguments "for" the proposition, and biased arguments "against" the proposition - that is not enough information to properly vote! With enough proper information, very often a bill will pass with an overwhelming majority for, or against, it - because everyone can clearly see through the smoke, and knows the reality of what is really going on, in a balanced way.

Should I encourage the schools to spend more money on lunches for students? I would, if that's the worst problem the school has. I would NOT if there are more serious problems preventing students from learning, such as a lack of teachers, rooms, safety, or when there are no books in the classroom! Problems are all relative, and the decision can only be made in relation to other related problems. Where is all the other information I need to properly vote on this? If I have to scrounge it all myself, spending hours and hours researching and calling officials during normal business hours - who has time to do that? And, every voting tax payer needs to do this! Without an easy-to-use centralized database, the right thing cannot happen.

The tax paying people have a right to know clearly what is going on in our government, and to a limited degree they can find out today. But the information is not collected in a central location, and there is no way to navigate the data easily. There's way too much disorganization and over-detail in some areas, with insufficient detail in other areas.

Proper data studies can lead to knowledge about how our tax money is being spent today, and this knowledge ultimately leads to the right use of money in the future. If a government agency is wasting money and I can't discover that because the information they're providing is too convoluted, incomplete and confusing, then the waste continues. If they have to provide all their spending in a clear chart for me to read, breaking everything down from large to small in a hierarchical tree of spending that I can easily navigate, I will see where they're wasting money, and I can call them on it - and work with them to reduce the waste.

There is always some waste in any system; I want to feel confident that we're keeping it at a minimum.

Shine the light anew, and the cockroaches will flee the area.

Data Visualization - A Good Solution

One way to make this information available is to show the first level breakdown of where our tax dollars go - how much goes to which overarching organizations within the country, during the past year or so. 5 or 6 top level bullets, with money that adds up to the total taken in by the IRS for that time period. It should be obvious that the numbers add up, and what the levels mean.

From there, the tax payer should be able to drill down - click on one organization to "open it"and see where all of its money is going, within its categories. Similarly, each category can be "opened" to see all the sub-categories of monetary expenditure. At each level, a clear English description should be given of what the money was spent on. If it can't be expressed clearly and simply, it should not have been spent in the first place. Again, the numbers all add up in this area, to the total in the level above.

The data should also have the ability for anybody to attach comments, at any level, to any piece of data or organization - a threaded discussion. Government officials post here to clarify any confusion or answer questions from tax payers about certain parts of the spending for that year. This meta-data would help future visitors to that part of the database, to reduce the number of redundant questions and confusion a specific piece of data could cause.

All data from all organizations needs to be regularly collected and fed to a central "public Internet server," and made available via a hierarchical viewer, as well as pure data feeds in a variety of formats: CSV (Excel), RSS (subscription), XML data dump, perhaps OData, and other formats that make sense. This can be automated, so it doesn't take much human work to keep the data flowing month after month, year after year.

Conclusion

It's time for our government to stop pretending incompetence, to stop staying behind the times. Computers are cheap and ubiquitous, as are databases and file systems with massive redundancy. The same for large-scale Internet access.

At any time any tax payer could access this database, and see where money really is being spent - what things are really being accomplished. This is when creativity kicks in - if we're doing X, why aren't we also doing Y? Are we spending too much/too little on Z? Why is organization A spending money on the same thing as organization B? Maybe it's right, or wrong; people can find out, contact those organizations, learn something, and update the comments in the database.

The direction of our country should be steerable by the people, at least to a certain extent. How can you do that when you can't tell what is actually being done, and how much money is spent each year doing it?

tl;dr

We need a Government database of spending, in tree-form, for us to play with.

I wanna explore, hate and love, what I find in it.

Best Humanitarian Use of Military Equipment

I laughed when I read this article. Someone figured out how to use old bomber airplanes to plant an unbelievable number of trees in one day. Check this out --

http://www.treehugger.com/files/2010/10/old-military-planes-drop-900000-tree-bombs-day.php

I hope they really do it. We've worried for so long about deforestation in many areas, it's awesome to think that treeless areas could be repaired at a rate of 3000 square miles per year, that's 1 billion trees in a year!