<article>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#article10_03_08_151248</id>
	<title>When the Power Goes Out At Google</title>
	<author>Soulskill</author>
	<datestamp>1268064420000</datestamp>
	<htmltext>1sockchuck writes <i>"What happens when the power goes out in one of Google's mighty data centers? The company has issued <a href="https://groups.google.com/group/google-appengine/browse\_thread/thread/a7640a2743922dcf?pli=1">an incident report on a Feb. 24 outage for Google App Engine</a>, which went offline when <a href="http://www.datacenterknowledge.com/archives/2010/03/08/when-the-power-goes-out-at-google/">an entire data center lost power</a>. The post-mortem outlines what went wrong and why, lessons learned and steps taken, which include additional training and documentation for staff and new datastore configurations for App Engine. Google is earning strong reviews for its openness, which is being <a href="http://www.transparentuptime.com/2010/03/google-app-engine-downtime-postmortem.html">hailed as an excellent model</a> for industry outage reports. At the other end of the spectrum is Australian host Datacom, where executives are denying that a Melbourne data center experienced water damage during weekend flooding, forcing tech media to <a href="http://www.itnews.com.au/News/169054,datacom-data-centre-flooded-by-melbourne-storm.aspx">document the outage</a> via photos, user stories and emails from the NOC."</i></htmltext>
<tokenext>1sockchuck writes " What happens when the power goes out in one of Google 's mighty data centers ?
The company has issued an incident report on a Feb. 24 outage for Google App Engine , which went offline when an entire data center lost power .
The post-mortem outlines what went wrong and why , lessons learned and steps taken , which include additional training and documentation for staff and new datastore configurations for App Engine .
Google is earning strong reviews for its openness , which is being hailed as an excellent model for industry outage reports .
At the other end of the spectrum is Australian host Datacom , where executives are denying that a Melbourne data center experienced water damage during weekend flooding , forcing tech media to document the outage via photos , user stories and emails from the NOC .
"</tokentext>
<sentencetext>1sockchuck writes "What happens when the power goes out in one of Google's mighty data centers?
The company has issued an incident report on a Feb. 24 outage for Google App Engine, which went offline when an entire data center lost power.
The post-mortem outlines what went wrong and why, lessons learned and steps taken, which include additional training and documentation for staff and new datastore configurations for App Engine.
Google is earning strong reviews for its openness, which is being hailed as an excellent model for industry outage reports.
At the other end of the spectrum is Australian host Datacom, where executives are denying that a Melbourne data center experienced water damage during weekend flooding, forcing tech media to document the outage via photos, user stories and emails from the NOC.
"</sentencetext>
</article>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31417960</id>
	<title>What happened</title>
	<author>Anonymous</author>
	<datestamp>1268167740000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>The local power utility accidentally tripped the only bus supplying the 30MW of power to the data center.</p><p>Google's data center lost all power for 8 seconds before the generators, and not all of them, came online.</p><p>The UPS system failed completely.</p><p>This facility is about 2 years old in SC.</p></htmltext>
<tokenext>The local power utility accidentally tripped the only bus supplying the 30MW of power to the data center.Google 's data center lost all power for 8 seconds before the generators , and not all of them , came online.The UPS system failed completely.This facility is about 2 years old in SC .</tokentext>
<sentencetext>The local power utility accidentally tripped the only bus supplying the 30MW of power to the data center.Google's data center lost all power for 8 seconds before the generators, and not all of them, came online.The UPS system failed completely.This facility is about 2 years old in SC.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402022</id>
	<title>Re:Read the comments</title>
	<author>houghi</author>
	<datestamp>1268071260000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>An interview with him from a previous 'non-event' : <a href="http://www.youtube.com/watch?v=WcU4t6zRAKg" title="youtube.com">http://www.youtube.com/watch?v=WcU4t6zRAKg</a> [youtube.com]</p></htmltext>
<tokenext>An interview with him from a previous 'non-event ' : http : //www.youtube.com/watch ? v = WcU4t6zRAKg [ youtube.com ]</tokentext>
<sentencetext>An interview with him from a previous 'non-event' : http://www.youtube.com/watch?v=WcU4t6zRAKg [youtube.com]</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402382</id>
	<title>Re:what about having people onsite?</title>
	<author>hedwards</author>
	<datestamp>1268072880000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>2</modscore>
	<htmltext>My parents once lost power for several hours because a crow got fried in one of the transformers down the street. People around here lose power from time to time when a tree falls on a line. Unplanned power outages are going to happen. Even though line reliability is probably higher now than at any time in the past, it still happens and companies like Google that rely upon it being always there should have plans.<br> <br>

This isn't just about keeping the people that use Google services informed, this is an admission that there's something to fix and that they're going to fix what they can. There isn't any particular reason why they need to disclose such plans beyond being a huge player and not wanting to scare away the numerous people that count on them for important work.</htmltext>
<tokenext>My parents once lost power for several hours because a crow got fried in one of the transformers down the street .
People around here lose power from time to time when a tree falls on a line .
Unplanned power outages are going to happen .
Even though line reliability is probably higher now than at any time in the past , it still happens and companies like Google that rely upon it being always there should have plans .
This is n't just about keeping the people that use Google services informed , this is an admission that there 's something to fix and that they 're going to fix what they can .
There is n't any particular reason why they need to disclose such plans beyond being a huge player and not wanting to scare away the numerous people that count on them for important work .</tokentext>
<sentencetext>My parents once lost power for several hours because a crow got fried in one of the transformers down the street.
People around here lose power from time to time when a tree falls on a line.
Unplanned power outages are going to happen.
Even though line reliability is probably higher now than at any time in the past, it still happens and companies like Google that rely upon it being always there should have plans.
This isn't just about keeping the people that use Google services informed, this is an admission that there's something to fix and that they're going to fix what they can.
There isn't any particular reason why they need to disclose such plans beyond being a huge player and not wanting to scare away the numerous people that count on them for important work.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401856</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402808</id>
	<title>Lucky they have multiple datacenters</title>
	<author>Anonymous</author>
	<datestamp>1268074680000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>Google is lucky they have a second, third, n number of datacenters to failover to in the first place. You might be surprised how many large companies still rely on truck-shipped tapes or other "cold" disaster recovery methods even for their most critical business data. If you had to restore your systems via tape, would your company still be alive by the time you came back up? Or would the negative publicity from the event lead to a slow and timely death? Although this was an eye-opening experience for Google, it should be even moreso for companies who haven't had to experience this type of an event. Unfortunately in my experience many companies (Google included) will not change disaster recovery policies (or many other IT policies for that matter) until a significant event has occurred. The question is, should you bet your business and be reactive, or protect your business and be proactive? In Google's case, they were able to be reactive and will come out alright. Many others probably wouldn't be so lucky. All-in-all I believe this will be a great learning experience for Google, and as a side-effect will hopefully direct more people to looking at cloud technology to protect their business from outages.</p></htmltext>
<tokenext>Google is lucky they have a second , third , n number of datacenters to failover to in the first place .
You might be surprised how many large companies still rely on truck-shipped tapes or other " cold " disaster recovery methods even for their most critical business data .
If you had to restore your systems via tape , would your company still be alive by the time you came back up ?
Or would the negative publicity from the event lead to a slow and timely death ?
Although this was an eye-opening experience for Google , it should be even moreso for companies who have n't had to experience this type of an event .
Unfortunately in my experience many companies ( Google included ) will not change disaster recovery policies ( or many other IT policies for that matter ) until a significant event has occurred .
The question is , should you bet your business and be reactive , or protect your business and be proactive ?
In Google 's case , they were able to be reactive and will come out alright .
Many others probably would n't be so lucky .
All-in-all I believe this will be a great learning experience for Google , and as a side-effect will hopefully direct more people to looking at cloud technology to protect their business from outages .</tokentext>
<sentencetext>Google is lucky they have a second, third, n number of datacenters to failover to in the first place.
You might be surprised how many large companies still rely on truck-shipped tapes or other "cold" disaster recovery methods even for their most critical business data.
If you had to restore your systems via tape, would your company still be alive by the time you came back up?
Or would the negative publicity from the event lead to a slow and timely death?
Although this was an eye-opening experience for Google, it should be even moreso for companies who haven't had to experience this type of an event.
Unfortunately in my experience many companies (Google included) will not change disaster recovery policies (or many other IT policies for that matter) until a significant event has occurred.
The question is, should you bet your business and be reactive, or protect your business and be proactive?
In Google's case, they were able to be reactive and will come out alright.
Many others probably wouldn't be so lucky.
All-in-all I believe this will be a great learning experience for Google, and as a side-effect will hopefully direct more people to looking at cloud technology to protect their business from outages.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31408764</id>
	<title>Re:Useless for large scale problems</title>
	<author>listentoreason</author>
	<datestamp>1268059020000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p><div class="quote"><p>Of COURSE there are people onsite. Most likely they have anywhere from a dozen to a hundred people onsite.</p></div><p>and as long as you're quiet and don't try to damage the control systems, you can move about freely and <a href="http://drexfiles.files.wordpress.com/2009/06/borg-ship-star-trek-first-contact-11.jpg" title="wordpress.com" rel="nofollow">they'll generally ignore you</a> [wordpress.com] </p></div>
	</htmltext>
<tokenext>Of COURSE there are people onsite .
Most likely they have anywhere from a dozen to a hundred people onsite.and as long as you 're quiet and do n't try to damage the control systems , you can move about freely and they 'll generally ignore you [ wordpress.com ]</tokentext>
<sentencetext>Of COURSE there are people onsite.
Most likely they have anywhere from a dozen to a hundred people onsite.and as long as you're quiet and don't try to damage the control systems, you can move about freely and they'll generally ignore you [wordpress.com] 
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402172</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401486</id>
	<title>Oh, my lifestream</title>
	<author>bigredradio</author>
	<datestamp>1268068500000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>My lifestream was interrupted and I didn't even notice! (see <a href="http://tech.slashdot.org/story/10/03/08/0024205/Time-To-Take-the-Internet-Seriously" title="slashdot.org">http://tech.slashdot.org/story/10/03/08/0024205/Time-To-Take-the-Internet-Seriously</a> [slashdot.org] for reference)</htmltext>
<tokenext>My lifestream was interrupted and I did n't even notice !
( see http : //tech.slashdot.org/story/10/03/08/0024205/Time-To-Take-the-Internet-Seriously [ slashdot.org ] for reference )</tokentext>
<sentencetext>My lifestream was interrupted and I didn't even notice!
(see http://tech.slashdot.org/story/10/03/08/0024205/Time-To-Take-the-Internet-Seriously [slashdot.org] for reference)</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405096</id>
	<title>Re:Generators plus UPS FTMFW</title>
	<author>DragonWriter</author>
	<datestamp>1268041920000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><blockquote><div><p>Any data center worth it's weight in dirt, must have UPS devices sufficient to power all servers plus all network and infrastructure equipment, as well as the HVAC systems too, for a minimum of at least 2 full hours on batteries, in case the backup generators have difficulty in getting started up and online.</p></div></blockquote><p>Google's setup appears to rely on the fact that they have redundant data centers, so failover to another data center addresses this problem. The problem here, as identified in their post-mortem, is that for training and other reasons, the fail over wasn't handled correctly.</p><p>Since there are sources of data center failure that having UPS + Generator backup won't help with at all, for something like this redundant data centers are essential <i>whether or not</i> you use UPS + Generator backup. Once you have redundant data centers, this problem should be solvable with failover. So, I think Google's general approach was reasonable from the start, as are their plans (detailed in the post-mortem) to address the failure by addressing the training and other issues which prevented failover plans from being properly executed.</p></div>
	</htmltext>
<tokenext>Any data center worth it 's weight in dirt , must have UPS devices sufficient to power all servers plus all network and infrastructure equipment , as well as the HVAC systems too , for a minimum of at least 2 full hours on batteries , in case the backup generators have difficulty in getting started up and online.Google 's setup appears to rely on the fact that they have redundant data centers , so failover to another data center addresses this problem .
The problem here , as identified in their post-mortem , is that for training and other reasons , the fail over was n't handled correctly.Since there are sources of data center failure that having UPS + Generator backup wo n't help with at all , for something like this redundant data centers are essential whether or not you use UPS + Generator backup .
Once you have redundant data centers , this problem should be solvable with failover .
So , I think Google 's general approach was reasonable from the start , as are their plans ( detailed in the post-mortem ) to address the failure by addressing the training and other issues which prevented failover plans from being properly executed .</tokentext>
<sentencetext>Any data center worth it's weight in dirt, must have UPS devices sufficient to power all servers plus all network and infrastructure equipment, as well as the HVAC systems too, for a minimum of at least 2 full hours on batteries, in case the backup generators have difficulty in getting started up and online.Google's setup appears to rely on the fact that they have redundant data centers, so failover to another data center addresses this problem.
The problem here, as identified in their post-mortem, is that for training and other reasons, the fail over wasn't handled correctly.Since there are sources of data center failure that having UPS + Generator backup won't help with at all, for something like this redundant data centers are essential whether or not you use UPS + Generator backup.
Once you have redundant data centers, this problem should be solvable with failover.
So, I think Google's general approach was reasonable from the start, as are their plans (detailed in the post-mortem) to address the failure by addressing the training and other issues which prevented failover plans from being properly executed.
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401462</id>
	<title>Nothing to see here people.</title>
	<author>Anonymous</author>
	<datestamp>1268068440000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>Google was open. What exactly is the issue?</p></htmltext>
<tokenext>Google was open .
What exactly is the issue ?</tokentext>
<sentencetext>Google was open.
What exactly is the issue?</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402788</id>
	<title>Re:Generators plus UPS FTMFW</title>
	<author>Darth\_brooks</author>
	<datestamp>1268074620000</datestamp>
	<modclass>Funny</modclass>
	<modscore>2</modscore>
	<htmltext><p>Yeah, and when the guys at the Jesus Christ of Datcenters that you describe have to do something like, say, switch from generator to utility power manually, and the document that details that process is 18 months old and refers to electrical panels that don't exist anymore, you get what you had here. A failure of fail-over procedures. If the lowliest help desk / operator can't at least understand the documentation you've written, then you've failed.</p><p>The only equipment failure listed is a "power failure." Granted, that can be as simple as "car hits a telephone pole and knocks out a chunk of the grid, leaving your office in the dark", which should be an easily survivable event. But how do you handle a failure like "50kva inline UPS shits the bed leaving nothing but a smoking chassis that no one wants to go anywhere near?" or "HVAC unit fails on christmas eve when only a skeleton staff is on duty and fills the raised floor with 8 inches of water, shorting everything within an inch of its life and making it impossible to bring any hosted services back online?"</p><p>There's nothing like a little bit of "we had no idea these three or four unrelated circumstances could happen simultaneously" disaster porn to make you realize that A. Outage / DR / fail-over planning is more than just throwing money at stuff (UPS's, generators, redundant lines, etc) and B. No matter how good your plan is, it will never be 100\% effective.</p></htmltext>
<tokenext>Yeah , and when the guys at the Jesus Christ of Datcenters that you describe have to do something like , say , switch from generator to utility power manually , and the document that details that process is 18 months old and refers to electrical panels that do n't exist anymore , you get what you had here .
A failure of fail-over procedures .
If the lowliest help desk / operator ca n't at least understand the documentation you 've written , then you 've failed.The only equipment failure listed is a " power failure .
" Granted , that can be as simple as " car hits a telephone pole and knocks out a chunk of the grid , leaving your office in the dark " , which should be an easily survivable event .
But how do you handle a failure like " 50kva inline UPS shits the bed leaving nothing but a smoking chassis that no one wants to go anywhere near ?
" or " HVAC unit fails on christmas eve when only a skeleton staff is on duty and fills the raised floor with 8 inches of water , shorting everything within an inch of its life and making it impossible to bring any hosted services back online ?
" There 's nothing like a little bit of " we had no idea these three or four unrelated circumstances could happen simultaneously " disaster porn to make you realize that A. Outage / DR / fail-over planning is more than just throwing money at stuff ( UPS 's , generators , redundant lines , etc ) and B. No matter how good your plan is , it will never be 100 \ % effective .</tokentext>
<sentencetext>Yeah, and when the guys at the Jesus Christ of Datcenters that you describe have to do something like, say, switch from generator to utility power manually, and the document that details that process is 18 months old and refers to electrical panels that don't exist anymore, you get what you had here.
A failure of fail-over procedures.
If the lowliest help desk / operator can't at least understand the documentation you've written, then you've failed.The only equipment failure listed is a "power failure.
" Granted, that can be as simple as "car hits a telephone pole and knocks out a chunk of the grid, leaving your office in the dark", which should be an easily survivable event.
But how do you handle a failure like "50kva inline UPS shits the bed leaving nothing but a smoking chassis that no one wants to go anywhere near?
" or "HVAC unit fails on christmas eve when only a skeleton staff is on duty and fills the raised floor with 8 inches of water, shorting everything within an inch of its life and making it impossible to bring any hosted services back online?
"There's nothing like a little bit of "we had no idea these three or four unrelated circumstances could happen simultaneously" disaster porn to make you realize that A. Outage / DR / fail-over planning is more than just throwing money at stuff (UPS's, generators, redundant lines, etc) and B. No matter how good your plan is, it will never be 100\% effective.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402306</id>
	<title>Re:Generators plus UPS FTMFW</title>
	<author>mjwalshe</author>
	<datestamp>1268072580000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>quite thers a comment somwhere else about how 356 main was highly regarded lol - if everything insn't running of the batteries 24/7 it aint a real datacentre.</htmltext>
<tokenext>quite thers a comment somwhere else about how 356 main was highly regarded lol - if everything ins n't running of the batteries 24/7 it aint a real datacentre .</tokentext>
<sentencetext>quite thers a comment somwhere else about how 356 main was highly regarded lol - if everything insn't running of the batteries 24/7 it aint a real datacentre.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31404786</id>
	<title>Re:Useless for large scale problems</title>
	<author>interkin3tic</author>
	<datestamp>1268040360000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p><div class="quote"><p>But what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them. Was I ever going to sue them? Probably not, maybe just ask for a break on that month's hosting or something.</p></div><p>You wouldn't but come on, you know how we Americans are.  <a href="http://www.gamespot.com/news/6184323.html?sid=6184323&amp;part=rss&amp;subj=6184323" title="gamespot.com">We sue when we can't play Halo for a few days</a> [gamespot.com].</p><p>Chances aren't bad that someone was looking for a lawsuit, heading it off at the pass had a chance to prevent some stupid lawsuits which would waste time and only benefit lawyers, possibly requiring some invasive, poorly thought-out court-ordered hinderance which would have slowed the recovery.</p></div>
	</htmltext>
<tokenext>But what pissed me off ( and why I do n't host with them anymore ) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them .
Was I ever going to sue them ?
Probably not , maybe just ask for a break on that month 's hosting or something.You would n't but come on , you know how we Americans are .
We sue when we ca n't play Halo for a few days [ gamespot.com ] .Chances are n't bad that someone was looking for a lawsuit , heading it off at the pass had a chance to prevent some stupid lawsuits which would waste time and only benefit lawyers , possibly requiring some invasive , poorly thought-out court-ordered hinderance which would have slowed the recovery .</tokentext>
<sentencetext>But what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them.
Was I ever going to sue them?
Probably not, maybe just ask for a break on that month's hosting or something.You wouldn't but come on, you know how we Americans are.
We sue when we can't play Halo for a few days [gamespot.com].Chances aren't bad that someone was looking for a lawsuit, heading it off at the pass had a chance to prevent some stupid lawsuits which would waste time and only benefit lawyers, possibly requiring some invasive, poorly thought-out court-ordered hinderance which would have slowed the recovery.
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402172</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401814</id>
	<title>ISO9001</title>
	<author>Anonymous</author>
	<datestamp>1268070180000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>1</modscore>
	<htmltext><p>This should be standard practice... It's like the good bits of ISO9001 with a bit more openness. When done right, ISO9001 is a good model to follow.</p></htmltext>
<tokenext>This should be standard practice... It 's like the good bits of ISO9001 with a bit more openness .
When done right , ISO9001 is a good model to follow .</tokentext>
<sentencetext>This should be standard practice... It's like the good bits of ISO9001 with a bit more openness.
When done right, ISO9001 is a good model to follow.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574</id>
	<title>Read the comments</title>
	<author>Anonymous</author>
	<datestamp>1268068920000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>5</modscore>
	<htmltext><p>I pity EvilMuppet. Guy is a tool. There are contractual agreements that are in place to prevent pictures, aka the "rules" but when the data center blatantly LIES they are breaking the trust and violating the agreement. Case Law exists where contracts can be violated when one accuses the other of violating said contract.</p><p>That's what happened. The data center was lying about what happened to avoid responsibility for the equipment it was being paid to host. Pictures were taken and are being used to prove the company did violate the trust of the contract.</p><p>You can argue the semantics and legality of it but if this goes to court the pictures will be admissible and the data center will lose.</p></htmltext>
<tokenext>I pity EvilMuppet .
Guy is a tool .
There are contractual agreements that are in place to prevent pictures , aka the " rules " but when the data center blatantly LIES they are breaking the trust and violating the agreement .
Case Law exists where contracts can be violated when one accuses the other of violating said contract.That 's what happened .
The data center was lying about what happened to avoid responsibility for the equipment it was being paid to host .
Pictures were taken and are being used to prove the company did violate the trust of the contract.You can argue the semantics and legality of it but if this goes to court the pictures will be admissible and the data center will lose .</tokentext>
<sentencetext>I pity EvilMuppet.
Guy is a tool.
There are contractual agreements that are in place to prevent pictures, aka the "rules" but when the data center blatantly LIES they are breaking the trust and violating the agreement.
Case Law exists where contracts can be violated when one accuses the other of violating said contract.That's what happened.
The data center was lying about what happened to avoid responsibility for the equipment it was being paid to host.
Pictures were taken and are being used to prove the company did violate the trust of the contract.You can argue the semantics and legality of it but if this goes to court the pictures will be admissible and the data center will lose.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402278</id>
	<title>Re:the worst nightmare of data center peeps</title>
	<author>mjwalshe</author>
	<datestamp>1268072460000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>switch to the alternate DC - I worked for BT and the set up an alternate DC across town for Telecom Gold just in case the thames flooded</htmltext>
<tokenext>switch to the alternate DC - I worked for BT and the set up an alternate DC across town for Telecom Gold just in case the thames flooded</tokentext>
<sentencetext>switch to the alternate DC - I worked for BT and the set up an alternate DC across town for Telecom Gold just in case the thames flooded</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31420650</id>
	<title>Re:what about having people onsite?</title>
	<author>RockWolf</author>
	<datestamp>1268136120000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p><div class="quote"><p>My parents once lost power for several hours because a crow got fried in one of the transformers...</p></div><p>Hah - payback for the LHC baguette incident. One-all now, Mother Nature!</p></div>
	</htmltext>
<tokenext>My parents once lost power for several hours because a crow got fried in one of the transformers...Hah - payback for the LHC baguette incident .
One-all now , Mother Nature !</tokentext>
<sentencetext>My parents once lost power for several hours because a crow got fried in one of the transformers...Hah - payback for the LHC baguette incident.
One-all now, Mother Nature!
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402382</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401998</id>
	<title>Re:title should read "Google App Engine NOT a Clou</title>
	<author>Davorama</author>
	<datestamp>1268071200000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>Sounds more like fog to me.</p></htmltext>
<tokenext>Sounds more like fog to me .</tokentext>
<sentencetext>Sounds more like fog to me.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401576</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403530</id>
	<title>Re:Read the comments</title>
	<author>sexconker</author>
	<datestamp>1268078040000</datestamp>
	<modclass>None</modclass>
	<modscore>-1</modscore>
	<htmltext><p>There is no such thing as case law.<br>There is legal precedent, which is not law.</p><p>Judges and lawyers who follow precedent are lazy, spineless fucks.</p><p>Each case should be decided upon according to the spirit and letter of the law.</p><p>"Case law" is not law that regulates people.<br>"Case law" is legalese for "someone else did the hard work before us, and until someone else less lazy than me comes along and argues against it, this is what we'll do - no need to rock the boat".</p><p>It is either law or precedent.<br>If it is law, it must be used to judge (both letter AND spirit).  It potentially can be nullified by the jury, or later changed by the legislative, but the lawyers and judge have no power over those processes beyond their individual vote.</p><p>If it is precedent, it is simply a tool to be used to speed up the legal process in similar cases.  It is not a means of "stabilizing" law.  It is not binding.  All cases are to be judged individually.  Our courts are based on the idea that judges fuck up - we have appeals.  It is folly to look at precedent as anything more than a program of what arguments each side will likely make and what laws are likely in play.</p></htmltext>
<tokenext>There is no such thing as case law.There is legal precedent , which is not law.Judges and lawyers who follow precedent are lazy , spineless fucks.Each case should be decided upon according to the spirit and letter of the law .
" Case law " is not law that regulates people .
" Case law " is legalese for " someone else did the hard work before us , and until someone else less lazy than me comes along and argues against it , this is what we 'll do - no need to rock the boat " .It is either law or precedent.If it is law , it must be used to judge ( both letter AND spirit ) .
It potentially can be nullified by the jury , or later changed by the legislative , but the lawyers and judge have no power over those processes beyond their individual vote.If it is precedent , it is simply a tool to be used to speed up the legal process in similar cases .
It is not a means of " stabilizing " law .
It is not binding .
All cases are to be judged individually .
Our courts are based on the idea that judges fuck up - we have appeals .
It is folly to look at precedent as anything more than a program of what arguments each side will likely make and what laws are likely in play .</tokentext>
<sentencetext>There is no such thing as case law.There is legal precedent, which is not law.Judges and lawyers who follow precedent are lazy, spineless fucks.Each case should be decided upon according to the spirit and letter of the law.
"Case law" is not law that regulates people.
"Case law" is legalese for "someone else did the hard work before us, and until someone else less lazy than me comes along and argues against it, this is what we'll do - no need to rock the boat".It is either law or precedent.If it is law, it must be used to judge (both letter AND spirit).
It potentially can be nullified by the jury, or later changed by the legislative, but the lawyers and judge have no power over those processes beyond their individual vote.If it is precedent, it is simply a tool to be used to speed up the legal process in similar cases.
It is not a means of "stabilizing" law.
It is not binding.
All cases are to be judged individually.
Our courts are based on the idea that judges fuck up - we have appeals.
It is folly to look at precedent as anything more than a program of what arguments each side will likely make and what laws are likely in play.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401512</id>
	<title>Isn't this part of their SLA?</title>
	<author>HerculesMO</author>
	<datestamp>1268068620000</datestamp>
	<modclass>Troll</modclass>
	<modscore>0</modscore>
	<htmltext><p>I thought that contracts required Google to disclose the cause and time of their downtime, and this disclosure is part of that.</p><p>Right now though, Google is making Microsoft look like they have better uptime for SaaS.</p></htmltext>
<tokenext>I thought that contracts required Google to disclose the cause and time of their downtime , and this disclosure is part of that.Right now though , Google is making Microsoft look like they have better uptime for SaaS .</tokentext>
<sentencetext>I thought that contracts required Google to disclose the cause and time of their downtime, and this disclosure is part of that.Right now though, Google is making Microsoft look like they have better uptime for SaaS.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401862</id>
	<title>"no online database will replace your daily news"</title>
	<author>SlappyBastard</author>
	<datestamp>1268070480000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>OMFG!  There's swinging at an outside pitch and there's try to hit one that was thrown in the fuckin' stands!!</htmltext>
<tokenext>OMFG !
There 's swinging at an outside pitch and there 's try to hit one that was thrown in the fuckin ' stands !
!</tokentext>
<sentencetext>OMFG!
There's swinging at an outside pitch and there's try to hit one that was thrown in the fuckin' stands!
!</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403724</id>
	<title>Re:Generators plus UPS FTMFW</title>
	<author>Pentium100</author>
	<datestamp>1268078820000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>Of course you can verify battery performance safely. My UPS has battery test (checks if the batteries can still be used, if it fails, batteries need replacing) and run time calibration (discharges batteries to 25\% and monitors how long it took, based on that it can estimate how long will it be able to hold the load). The whatever system google is using should be able to check the batteries while power is on, so that you don't end up with batteries that have 20\% of their original capacity when the power goes down.</p></htmltext>
<tokenext>Of course you can verify battery performance safely .
My UPS has battery test ( checks if the batteries can still be used , if it fails , batteries need replacing ) and run time calibration ( discharges batteries to 25 \ % and monitors how long it took , based on that it can estimate how long will it be able to hold the load ) .
The whatever system google is using should be able to check the batteries while power is on , so that you do n't end up with batteries that have 20 \ % of their original capacity when the power goes down .</tokentext>
<sentencetext>Of course you can verify battery performance safely.
My UPS has battery test (checks if the batteries can still be used, if it fails, batteries need replacing) and run time calibration (discharges batteries to 25\% and monitors how long it took, based on that it can estimate how long will it be able to hold the load).
The whatever system google is using should be able to check the batteries while power is on, so that you don't end up with batteries that have 20\% of their original capacity when the power goes down.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402350</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403272</id>
	<title>floods</title>
	<author>zogger</author>
	<datestamp>1268076840000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>2</modscore>
	<htmltext><p>Did you ever actually see a big flood? Freaking awesome power, like a fleet of bulldozers. Smashes stuff, rips houses off foundations, knocks huge trees over, will tumble multiple ton boulders ahead of it, etc. Just depends on how big the flood is. We had one late last year here, six inches of rain in a couple of hours, just tore stuff up all over. The "building" that can withstand a flood of significant size exists, it is called a submarine. Most buildings of the normal kind just aren't designed to deal with anything that destructive. Some can resist minor floods, but not too many.</p></htmltext>
<tokenext>Did you ever actually see a big flood ?
Freaking awesome power , like a fleet of bulldozers .
Smashes stuff , rips houses off foundations , knocks huge trees over , will tumble multiple ton boulders ahead of it , etc .
Just depends on how big the flood is .
We had one late last year here , six inches of rain in a couple of hours , just tore stuff up all over .
The " building " that can withstand a flood of significant size exists , it is called a submarine .
Most buildings of the normal kind just are n't designed to deal with anything that destructive .
Some can resist minor floods , but not too many .</tokentext>
<sentencetext>Did you ever actually see a big flood?
Freaking awesome power, like a fleet of bulldozers.
Smashes stuff, rips houses off foundations, knocks huge trees over, will tumble multiple ton boulders ahead of it, etc.
Just depends on how big the flood is.
We had one late last year here, six inches of rain in a couple of hours, just tore stuff up all over.
The "building" that can withstand a flood of significant size exists, it is called a submarine.
Most buildings of the normal kind just aren't designed to deal with anything that destructive.
Some can resist minor floods, but not too many.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402510</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402516</id>
	<title>Re:the worst nightmare of data center peeps</title>
	<author>Anonymous</author>
	<datestamp>1268073360000</datestamp>
	<modclass>Troll</modclass>
	<modscore>-1</modscore>
	<htmltext><p><div class="quote"><p>I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.</p></div><p>Then you are doing a rubbish job at your job! I would never hire somebody who would't ask this question up front before making a hosting decision, and having this decision made before you joined is no excuse. You are another typical slashdot failure of a sysadmin. Disaster recovery IS a sysadmins primary job. If your network cannot handle a disaster like this, then your network is rubbish.</p></div>
	</htmltext>
<tokenext>I 've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical ( including the dual redundant UPS systems ) in the data center.Then you are doing a rubbish job at your job !
I would never hire somebody who would't ask this question up front before making a hosting decision , and having this decision made before you joined is no excuse .
You are another typical slashdot failure of a sysadmin .
Disaster recovery IS a sysadmins primary job .
If your network can not handle a disaster like this , then your network is rubbish .</tokentext>
<sentencetext>I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.Then you are doing a rubbish job at your job!
I would never hire somebody who would't ask this question up front before making a hosting decision, and having this decision made before you joined is no excuse.
You are another typical slashdot failure of a sysadmin.
Disaster recovery IS a sysadmins primary job.
If your network cannot handle a disaster like this, then your network is rubbish.
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402066</id>
	<title>When the Power Goes Out At Google...</title>
	<author>binaryseraph</author>
	<datestamp>1268071440000</datestamp>
	<modclass>Informativ</modclass>
	<modscore>3</modscore>
	<htmltext>...a fairy dies.</htmltext>
<tokenext>...a fairy dies .</tokentext>
<sentencetext>...a fairy dies.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405204</id>
	<title>Outside or the TV</title>
	<author>Anonymous</author>
	<datestamp>1268042400000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>I turn on my TV or go outside to return to a normal life.</p></htmltext>
<tokenext>I turn on my TV or go outside to return to a normal life .</tokentext>
<sentencetext>I turn on my TV or go outside to return to a normal life.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480</id>
	<title>what about having people onsite?</title>
	<author>Anonymous</author>
	<datestamp>1268068500000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>2</modscore>
	<htmltext><p>aren't there any people in the data center to tell them that yes there has been a power outage, so and so machines are affected, etc? sounds like all they have is remote monitoring and if something happens than someone has to drive to the location to see what's wrong</p></htmltext>
<tokenext>are n't there any people in the data center to tell them that yes there has been a power outage , so and so machines are affected , etc ?
sounds like all they have is remote monitoring and if something happens than someone has to drive to the location to see what 's wrong</tokentext>
<sentencetext>aren't there any people in the data center to tell them that yes there has been a power outage, so and so machines are affected, etc?
sounds like all they have is remote monitoring and if something happens than someone has to drive to the location to see what's wrong</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403138</id>
	<title>Lessons Like</title>
	<author>Greyfox</author>
	<datestamp>1268076360000</datestamp>
	<modclass>Funny</modclass>
	<modscore>2</modscore>
	<htmltext>Don't have all your shit in one data center, maybe? I'd have thought that one would be pretty fundamental. Of course, knowing Google they're going to decide that what they really need is power generation right on site, then they'll just pop off and invent nuclear fusion before lunch.</htmltext>
<tokenext>Do n't have all your shit in one data center , maybe ?
I 'd have thought that one would be pretty fundamental .
Of course , knowing Google they 're going to decide that what they really need is power generation right on site , then they 'll just pop off and invent nuclear fusion before lunch .</tokentext>
<sentencetext>Don't have all your shit in one data center, maybe?
I'd have thought that one would be pretty fundamental.
Of course, knowing Google they're going to decide that what they really need is power generation right on site, then they'll just pop off and invent nuclear fusion before lunch.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401506</id>
	<title>An "Incident"?</title>
	<author>No Lucifer</author>
	<datestamp>1268068560000</datestamp>
	<modclass>Offtopic</modclass>
	<modscore>0</modscore>
	<htmltext>Jack must've forgotten to enter the code...</htmltext>
<tokenext>Jack must 've forgotten to enter the code.. .</tokentext>
<sentencetext>Jack must've forgotten to enter the code...</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402350</id>
	<title>Re:Generators plus UPS FTMFW</title>
	<author>Anonymous</author>
	<datestamp>1268072760000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>Two full hours requires a MASSIVE battery capacity. It's far more feasible to count on 10-15 minutes from the batteries and make sure your generators start up promptly.</p><p>Also, some of Google's datacenters (not sure if this is one of them) dispense with many centralized batteries in favor of building the battery into each server alongside the PSU. This avoids some issues with AC-&gt;DC-&gt;AC conversion, leaving them with just AC-&gt;DC at each server. I'm just speculating, but it's possible that generator startup went as planned and the 25\% of servers that didn't survive the outage turned out to have too-short battery life on their local battery packs. Hard to verify battery performance without a live test...</p></htmltext>
<tokenext>Two full hours requires a MASSIVE battery capacity .
It 's far more feasible to count on 10-15 minutes from the batteries and make sure your generators start up promptly.Also , some of Google 's datacenters ( not sure if this is one of them ) dispense with many centralized batteries in favor of building the battery into each server alongside the PSU .
This avoids some issues with AC- &gt; DC- &gt; AC conversion , leaving them with just AC- &gt; DC at each server .
I 'm just speculating , but it 's possible that generator startup went as planned and the 25 \ % of servers that did n't survive the outage turned out to have too-short battery life on their local battery packs .
Hard to verify battery performance without a live test.. .</tokentext>
<sentencetext>Two full hours requires a MASSIVE battery capacity.
It's far more feasible to count on 10-15 minutes from the batteries and make sure your generators start up promptly.Also, some of Google's datacenters (not sure if this is one of them) dispense with many centralized batteries in favor of building the battery into each server alongside the PSU.
This avoids some issues with AC-&gt;DC-&gt;AC conversion, leaving them with just AC-&gt;DC at each server.
I'm just speculating, but it's possible that generator startup went as planned and the 25\% of servers that didn't survive the outage turned out to have too-short battery life on their local battery packs.
Hard to verify battery performance without a live test...</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401478</id>
	<title>Really Quiet</title>
	<author>Anonymous</author>
	<datestamp>1268068500000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>It gets really quite.</p></htmltext>
<tokenext>It gets really quite .</tokentext>
<sentencetext>It gets really quite.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403654</id>
	<title>Re:title should read "Google App Engine NOT a Clou</title>
	<author>RoFLKOPTr</author>
	<datestamp>1268078520000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p><div class="quote"><p>Obviously if the power goes out, and the service goes offline, then it WASN'T a cloud.  If it's a cloud, it can't go down.  If it goes down, it wasn't a cloud.</p></div><p>The cloud got too big and it rained.</p></div>
	</htmltext>
<tokenext>Obviously if the power goes out , and the service goes offline , then it WAS N'T a cloud .
If it 's a cloud , it ca n't go down .
If it goes down , it was n't a cloud.The cloud got too big and it rained .</tokentext>
<sentencetext>Obviously if the power goes out, and the service goes offline, then it WASN'T a cloud.
If it's a cloud, it can't go down.
If it goes down, it wasn't a cloud.The cloud got too big and it rained.
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401576</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401636</id>
	<title>They had a perfect contingency plan for this case</title>
	<author>juanjux</author>
	<datestamp>1268069280000</datestamp>
	<modclass>Funny</modclass>
	<modscore>5</modscore>
	<htmltext><p>...but it was stored on Google Docs.</p></htmltext>
<tokenext>...but it was stored on Google Docs .</tokentext>
<sentencetext>...but it was stored on Google Docs.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405894</id>
	<title>Post Mortem Missed the Problem</title>
	<author>photonrider</author>
	<datestamp>1268045280000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>I read the post-mortem and I think they completely missed the mark. Power failed to some machines. They only noticed because "...traffic has problems..." They should have been monitoring the power to detect this situation.

They didn't say whether they have the data center power supply on a UPS or not. If it was, it was dying and no one noticed.

If they had been monitoring the power they might have avoided the whole mess.</htmltext>
<tokenext>I read the post-mortem and I think they completely missed the mark .
Power failed to some machines .
They only noticed because " ...traffic has problems... " They should have been monitoring the power to detect this situation .
They did n't say whether they have the data center power supply on a UPS or not .
If it was , it was dying and no one noticed .
If they had been monitoring the power they might have avoided the whole mess .</tokentext>
<sentencetext>I read the post-mortem and I think they completely missed the mark.
Power failed to some machines.
They only noticed because "...traffic has problems..." They should have been monitoring the power to detect this situation.
They didn't say whether they have the data center power supply on a UPS or not.
If it was, it was dying and no one noticed.
If they had been monitoring the power they might have avoided the whole mess.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403282</id>
	<title>Back around 2005...</title>
	<author>kilodelta</author>
	<datestamp>1268076840000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>We decided to move three of our divisions into one facility, those included to business facing units and the I.T. division.
<br> <br>
I was charged with laying out the design for data, telecom and electrical for the project. Also had engineering of our little NOC.
<br> <br>
Nice setup - redundant power in the I.T. division, nice big APC UPS for the entire room, had it's own 480V power drop, dual HVAC units, a natural gas fired generator. It's nice to have the money to do this.
<br> <br>
Since we were a state agency we had to use state DNS services. And one day the city had a massive power outage. We were up and running happy as a clam but we found the Achilles heel in all our plans. Without DNS we couldn't get in or out. I had floated the idea of maintaining our own DNS server but nobody wanted to hear that. We had the decent network connection, and the redundant power (Yes, we even placed a UPS/Generator backed up outlet in the MDF for Cox's Marconi router) so why the hell not replicate the state DNS services?
<br> <br>
Let that be a lesson. We tried to plan for all contingencies and we completely missed our dependence on an outside state agency. Of course since a river runs right behind we also raised the NOC floor by about a foot.</htmltext>
<tokenext>We decided to move three of our divisions into one facility , those included to business facing units and the I.T .
division . I was charged with laying out the design for data , telecom and electrical for the project .
Also had engineering of our little NOC .
Nice setup - redundant power in the I.T .
division , nice big APC UPS for the entire room , had it 's own 480V power drop , dual HVAC units , a natural gas fired generator .
It 's nice to have the money to do this .
Since we were a state agency we had to use state DNS services .
And one day the city had a massive power outage .
We were up and running happy as a clam but we found the Achilles heel in all our plans .
Without DNS we could n't get in or out .
I had floated the idea of maintaining our own DNS server but nobody wanted to hear that .
We had the decent network connection , and the redundant power ( Yes , we even placed a UPS/Generator backed up outlet in the MDF for Cox 's Marconi router ) so why the hell not replicate the state DNS services ?
Let that be a lesson .
We tried to plan for all contingencies and we completely missed our dependence on an outside state agency .
Of course since a river runs right behind we also raised the NOC floor by about a foot .</tokentext>
<sentencetext>We decided to move three of our divisions into one facility, those included to business facing units and the I.T.
division.
 
I was charged with laying out the design for data, telecom and electrical for the project.
Also had engineering of our little NOC.
Nice setup - redundant power in the I.T.
division, nice big APC UPS for the entire room, had it's own 480V power drop, dual HVAC units, a natural gas fired generator.
It's nice to have the money to do this.
Since we were a state agency we had to use state DNS services.
And one day the city had a massive power outage.
We were up and running happy as a clam but we found the Achilles heel in all our plans.
Without DNS we couldn't get in or out.
I had floated the idea of maintaining our own DNS server but nobody wanted to hear that.
We had the decent network connection, and the redundant power (Yes, we even placed a UPS/Generator backed up outlet in the MDF for Cox's Marconi router) so why the hell not replicate the state DNS services?
Let that be a lesson.
We tried to plan for all contingencies and we completely missed our dependence on an outside state agency.
Of course since a river runs right behind we also raised the NOC floor by about a foot.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401856</id>
	<title>Re:what about having people onsite?</title>
	<author>dch24</author>
	<datestamp>1268070480000</datestamp>
	<modclass>Troll</modclass>
	<modscore>1</modscore>
	<htmltext>What I want to know is, what caused the outage?<br>
<br>
The post on the google-appengine group details all the things they did wrong and are going to fix, <b>after the power went out</b>. Fine, I have to plan for outages too. But what caused the unplanned outage?</htmltext>
<tokenext>What I want to know is , what caused the outage ?
The post on the google-appengine group details all the things they did wrong and are going to fix , after the power went out .
Fine , I have to plan for outages too .
But what caused the unplanned outage ?</tokentext>
<sentencetext>What I want to know is, what caused the outage?
The post on the google-appengine group details all the things they did wrong and are going to fix, after the power went out.
Fine, I have to plan for outages too.
But what caused the unplanned outage?</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402210</id>
	<title>Has anyone from Ubisoft read this?</title>
	<author>Ben4jammin</author>
	<datestamp>1268072220000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext>I think it would do them good, considering the recent downtime with Assassin's Creed 2.  Has anyone seen any info on that outage?</htmltext>
<tokenext>I think it would do them good , considering the recent downtime with Assassin 's Creed 2 .
Has anyone seen any info on that outage ?</tokentext>
<sentencetext>I think it would do them good, considering the recent downtime with Assassin's Creed 2.
Has anyone seen any info on that outage?</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401528</id>
	<title>and what about openess during the incident?</title>
	<author>alen</author>
	<datestamp>1268068680000</datestamp>
	<modclass>None</modclass>
	<modscore>-1</modscore>
	<htmltext><p>if i called Google support during the incident, would i have been told the truth. Or would they have told me that everything is fine and to check my end</p></htmltext>
<tokenext>if i called Google support during the incident , would i have been told the truth .
Or would they have told me that everything is fine and to check my end</tokentext>
<sentencetext>if i called Google support during the incident, would i have been told the truth.
Or would they have told me that everything is fine and to check my end</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402172</id>
	<title>Useless for large scale problems</title>
	<author>mcrbids</author>
	<datestamp>1268071920000</datestamp>
	<modclass>Interestin</modclass>
	<modscore>5</modscore>
	<htmltext><p>Of COURSE there are people onsite. Most likely they have anywhere from a dozen to a hundred people onsite. But what's that going to do for you in the case of a large-scale problem?</p><p>The otherwise top rated <a href="http://www.365main.com/" title="365main.com">365 Main </a> [365main.com] facility in San Francisco went down a few years ago. They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works. Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out. It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.</p><p>So there you are, with perhaps 25 staff a 4-story building with tens of thousands of servers, the power is out, nobody can figure out why, and the phone lines are so loaded it's worthless. Even when the power comes back on, it's not like you are going to get "hot hands" in anything less than a week!</p><p>Hey, even with all the best planning, disasters like this DO happen! I had to spend 2 wracking days driving to S.F. (several hours drive) to witness a disaster zone. HUNDREDS of techs just like myself carefully nursing their servers back to health, running disk checks, talking in tense tones on cell phones, etc.</p><p>But <b>what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them.</b> Was I ever going to sue them? Probably not, maybe just ask for a break on that month's hosting or something. I mean, I just want the damned stuff to work, and I appreciate that even in the best of situations, things *can* go wrong.</p><p>So now I host with <a href="heraklesdata.com" title="slashdot.org">Herakles data center</a> [slashdot.org] which is just as nice as the S.F. facility, except that it's closer, and it's even noticably cheaper. Redundant power, redundant network feeds, just like 365 main. (Better: they had redundancy all the way into my cage, 365 Main just had redundancy to the cage's main power feed)</p><p>And, after a year or two of hosting with Herakles, they had a "brown-out" situation, where one of their main Cisco routers went partially dark, working well enough that their redundant router didn't kick in right away, leaving some routes up and others down while they tried to figure out what was going on.</p><p>When all was said and done, they simply sent out a statement of "Here's what happened, it violates some of your TOS agreements, and here's a claim form". It was so nice, and so open, that out of sheer goodwill, I didn't bother to fill out a claim form, and can't praise them highly enough!</p></htmltext>
<tokenext>Of COURSE there are people onsite .
Most likely they have anywhere from a dozen to a hundred people onsite .
But what 's that going to do for you in the case of a large-scale problem ? The otherwise top rated 365 Main [ 365main.com ] facility in San Francisco went down a few years ago .
They had all the shizz , multipoint redundant power , multiple data feeds , earthquake-resistant building , the works .
Yet , their equipment was n't well equipped to handle what actually took them down - a recurring brown-out .
It confused their equipment , which failed to " see " the situation as one requiring emergency power , causing the whole building to go dark.So there you are , with perhaps 25 staff a 4-story building with tens of thousands of servers , the power is out , nobody can figure out why , and the phone lines are so loaded it 's worthless .
Even when the power comes back on , it 's not like you are going to get " hot hands " in anything less than a week ! Hey , even with all the best planning , disasters like this DO happen !
I had to spend 2 wracking days driving to S.F .
( several hours drive ) to witness a disaster zone .
HUNDREDS of techs just like myself carefully nursing their servers back to health , running disk checks , talking in tense tones on cell phones , etc.But what pissed me off ( and why I do n't host with them anymore ) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them .
Was I ever going to sue them ?
Probably not , maybe just ask for a break on that month 's hosting or something .
I mean , I just want the damned stuff to work , and I appreciate that even in the best of situations , things * can * go wrong.So now I host with Herakles data center [ slashdot.org ] which is just as nice as the S.F .
facility , except that it 's closer , and it 's even noticably cheaper .
Redundant power , redundant network feeds , just like 365 main .
( Better : they had redundancy all the way into my cage , 365 Main just had redundancy to the cage 's main power feed ) And , after a year or two of hosting with Herakles , they had a " brown-out " situation , where one of their main Cisco routers went partially dark , working well enough that their redundant router did n't kick in right away , leaving some routes up and others down while they tried to figure out what was going on.When all was said and done , they simply sent out a statement of " Here 's what happened , it violates some of your TOS agreements , and here 's a claim form " .
It was so nice , and so open , that out of sheer goodwill , I did n't bother to fill out a claim form , and ca n't praise them highly enough !</tokentext>
<sentencetext>Of COURSE there are people onsite.
Most likely they have anywhere from a dozen to a hundred people onsite.
But what's that going to do for you in the case of a large-scale problem?The otherwise top rated 365 Main  [365main.com] facility in San Francisco went down a few years ago.
They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works.
Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out.
It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.So there you are, with perhaps 25 staff a 4-story building with tens of thousands of servers, the power is out, nobody can figure out why, and the phone lines are so loaded it's worthless.
Even when the power comes back on, it's not like you are going to get "hot hands" in anything less than a week!Hey, even with all the best planning, disasters like this DO happen!
I had to spend 2 wracking days driving to S.F.
(several hours drive) to witness a disaster zone.
HUNDREDS of techs just like myself carefully nursing their servers back to health, running disk checks, talking in tense tones on cell phones, etc.But what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them.
Was I ever going to sue them?
Probably not, maybe just ask for a break on that month's hosting or something.
I mean, I just want the damned stuff to work, and I appreciate that even in the best of situations, things *can* go wrong.So now I host with Herakles data center [slashdot.org] which is just as nice as the S.F.
facility, except that it's closer, and it's even noticably cheaper.
Redundant power, redundant network feeds, just like 365 main.
(Better: they had redundancy all the way into my cage, 365 Main just had redundancy to the cage's main power feed)And, after a year or two of hosting with Herakles, they had a "brown-out" situation, where one of their main Cisco routers went partially dark, working well enough that their redundant router didn't kick in right away, leaving some routes up and others down while they tried to figure out what was going on.When all was said and done, they simply sent out a statement of "Here's what happened, it violates some of your TOS agreements, and here's a claim form".
It was so nice, and so open, that out of sheer goodwill, I didn't bother to fill out a claim form, and can't praise them highly enough!</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31453156</id>
	<title>Re:Read the comments</title>
	<author>Anonymous</author>
	<datestamp>1268416980000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>Plus he makes two contradictory arguments.</p><p>First, he states that since you're not allowed photography in the datacentre, the actions of the person posting the images have meant that he and his colleagues:</p><p>"...had to make major changes to their work practices (and a number of us have had major production-level impacts as a result) due to no mobiles being allowed on the data floor."</p><p>Now, either the contract he's so obsessive about the observance of contains a clause forbidding photographic equipment of any kind on the floor (inc. mobiles with cameras) or it doesn't, and in the case of the latter, there's no good reason why he can't take his mobile into the building with him - unless he's abiding by contracts that don't even exist now.</p></htmltext>
<tokenext>Plus he makes two contradictory arguments.First , he states that since you 're not allowed photography in the datacentre , the actions of the person posting the images have meant that he and his colleagues : " ...had to make major changes to their work practices ( and a number of us have had major production-level impacts as a result ) due to no mobiles being allowed on the data floor .
" Now , either the contract he 's so obsessive about the observance of contains a clause forbidding photographic equipment of any kind on the floor ( inc. mobiles with cameras ) or it does n't , and in the case of the latter , there 's no good reason why he ca n't take his mobile into the building with him - unless he 's abiding by contracts that do n't even exist now .</tokentext>
<sentencetext>Plus he makes two contradictory arguments.First, he states that since you're not allowed photography in the datacentre, the actions of the person posting the images have meant that he and his colleagues:"...had to make major changes to their work practices (and a number of us have had major production-level impacts as a result) due to no mobiles being allowed on the data floor.
"Now, either the contract he's so obsessive about the observance of contains a clause forbidding photographic equipment of any kind on the floor (inc. mobiles with cameras) or it doesn't, and in the case of the latter, there's no good reason why he can't take his mobile into the building with him - unless he's abiding by contracts that don't even exist now.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31410828</id>
	<title>Re:Read the comments</title>
	<author>Th3 3vil Mupp3t</author>
	<datestamp>1268166540000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>Looking over the contract we have with Datacom, you'd be hard pressed to have the Managing Director's statements be material in affecting a contract violation. Given that the photos were taken well before any statement was made to the public by a Datacom representative takes at least some of the basis away from your argument of trust.</p><p>As for evidence, colleagues of mine have damaged equipment and I have remote monitoring, MRTG graphs and other means of validating facts. How do you think that a particular public statement can absolve a provider of responsibility or compliance to a contract precisely?</p><p>The pictures being admissible is another matter altogether - one I'm definitely not qualified to speculate on!</p><p>I'm definitely not employed by Datacom, and the fact that I've had to alter my work practices based entirely on photos such as these being published in the past is part of the basis for my contention regarding the issue of these specific photographs being taken. I'm very much not alone in this.</p></htmltext>
<tokenext>Looking over the contract we have with Datacom , you 'd be hard pressed to have the Managing Director 's statements be material in affecting a contract violation .
Given that the photos were taken well before any statement was made to the public by a Datacom representative takes at least some of the basis away from your argument of trust.As for evidence , colleagues of mine have damaged equipment and I have remote monitoring , MRTG graphs and other means of validating facts .
How do you think that a particular public statement can absolve a provider of responsibility or compliance to a contract precisely ? The pictures being admissible is another matter altogether - one I 'm definitely not qualified to speculate on ! I 'm definitely not employed by Datacom , and the fact that I 've had to alter my work practices based entirely on photos such as these being published in the past is part of the basis for my contention regarding the issue of these specific photographs being taken .
I 'm very much not alone in this .</tokentext>
<sentencetext>Looking over the contract we have with Datacom, you'd be hard pressed to have the Managing Director's statements be material in affecting a contract violation.
Given that the photos were taken well before any statement was made to the public by a Datacom representative takes at least some of the basis away from your argument of trust.As for evidence, colleagues of mine have damaged equipment and I have remote monitoring, MRTG graphs and other means of validating facts.
How do you think that a particular public statement can absolve a provider of responsibility or compliance to a contract precisely?The pictures being admissible is another matter altogether - one I'm definitely not qualified to speculate on!I'm definitely not employed by Datacom, and the fact that I've had to alter my work practices based entirely on photos such as these being published in the past is part of the basis for my contention regarding the issue of these specific photographs being taken.
I'm very much not alone in this.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31408340</id>
	<title>The Datacom response model</title>
	<author>puppet10</author>
	<datestamp>1268056080000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>Repeat to yourself: "All is well, All is well, All is well" and everything will be exactly like you wish it to be.</p><p>Note originators of response model are not responsible for anyone being taken away to a psychiatric facility because of a belief response model user is psychotic</p></htmltext>
<tokenext>Repeat to yourself : " All is well , All is well , All is well " and everything will be exactly like you wish it to be.Note originators of response model are not responsible for anyone being taken away to a psychiatric facility because of a belief response model user is psychotic</tokentext>
<sentencetext>Repeat to yourself: "All is well, All is well, All is well" and everything will be exactly like you wish it to be.Note originators of response model are not responsible for anyone being taken away to a psychiatric facility because of a belief response model user is psychotic</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402656</id>
	<title>Re:Generators plus UPS FTMFW</title>
	<author>Tynin</author>
	<datestamp>1268073960000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>3</modscore>
	<htmltext>You are so cute. I know very little about UPS systems, but when I was working in a datacenter that housed 5000 servers we had a two story room that was twice the size of most houses (~2000 sq ft) with rows and rows of batteries. I was told that in the event of a power outage, we had 22 minutes of battery power before everything went out. The idea of having enough for 2 hours would have been one an interesting setup considering how monstrously large this one already was. Besides, I'm unsure why you'd ever need more than that 22min since that is plenty of time for our on site staff to gracefully power down any of our major servers if the backup generator failed to kick in.</htmltext>
<tokenext>You are so cute .
I know very little about UPS systems , but when I was working in a datacenter that housed 5000 servers we had a two story room that was twice the size of most houses ( ~ 2000 sq ft ) with rows and rows of batteries .
I was told that in the event of a power outage , we had 22 minutes of battery power before everything went out .
The idea of having enough for 2 hours would have been one an interesting setup considering how monstrously large this one already was .
Besides , I 'm unsure why you 'd ever need more than that 22min since that is plenty of time for our on site staff to gracefully power down any of our major servers if the backup generator failed to kick in .</tokentext>
<sentencetext>You are so cute.
I know very little about UPS systems, but when I was working in a datacenter that housed 5000 servers we had a two story room that was twice the size of most houses (~2000 sq ft) with rows and rows of batteries.
I was told that in the event of a power outage, we had 22 minutes of battery power before everything went out.
The idea of having enough for 2 hours would have been one an interesting setup considering how monstrously large this one already was.
Besides, I'm unsure why you'd ever need more than that 22min since that is plenty of time for our on site staff to gracefully power down any of our major servers if the backup generator failed to kick in.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405394</id>
	<title>Re:Significantly higher latency?</title>
	<author>DragonWriter</author>
	<datestamp>1268043240000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><blockquote><div><p>Anyone know some numbers around what "significantly higher latency" means?</p></div></blockquote><p>I suspect not, since the feature hasn't been implemented yet.</p></div>
	</htmltext>
<tokenext>Anyone know some numbers around what " significantly higher latency " means ? I suspect not , since the feature has n't been implemented yet .</tokentext>
<sentencetext>Anyone know some numbers around what "significantly higher latency" means?I suspect not, since the feature hasn't been implemented yet.
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401700</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401700</id>
	<title>Significantly higher latency?</title>
	<author>nacturation</author>
	<datestamp>1268069580000</datestamp>
	<modclass>Interestin</modclass>
	<modscore>2</modscore>
	<htmltext><p><div class="quote"><p>A new option for higher availability using synchronous replication for reads and writes, at the cost of significantly higher latency</p> </div><p>Anyone know some numbers around what "significantly higher latency" means?  The <a href="http://code.google.com/status/appengine/detail/datastore/2010/03/07#ae-trust-detail-datastore-put-latency" title="google.com">current performance</a> [google.com] looks to be about 200ms on average.  Assuming this higher availability model doesn't commit a DB transaction until it's written to two separate datacenters, is this around 300 - 400ms for each put to the datastore?</p></div>
	</htmltext>
<tokenext>A new option for higher availability using synchronous replication for reads and writes , at the cost of significantly higher latency Anyone know some numbers around what " significantly higher latency " means ?
The current performance [ google.com ] looks to be about 200ms on average .
Assuming this higher availability model does n't commit a DB transaction until it 's written to two separate datacenters , is this around 300 - 400ms for each put to the datastore ?</tokentext>
<sentencetext>A new option for higher availability using synchronous replication for reads and writes, at the cost of significantly higher latency Anyone know some numbers around what "significantly higher latency" means?
The current performance [google.com] looks to be about 200ms on average.
Assuming this higher availability model doesn't commit a DB transaction until it's written to two separate datacenters, is this around 300 - 400ms for each put to the datastore?
	</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31411388</id>
	<title>Re:the worst nightmare of data center peeps</title>
	<author>Anonymous</author>
	<datestamp>1268132400000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>Not necessarily hard, but try steering clear of  <a href="http://www.theregister.co.uk/2009/04/30/google\_data\_center\_navy\_patent/" title="theregister.co.uk" rel="nofollow">patents</a> [theregister.co.uk]</p></htmltext>
<tokenext>Not necessarily hard , but try steering clear of patents [ theregister.co.uk ]</tokentext>
<sentencetext>Not necessarily hard, but try steering clear of  patents [theregister.co.uk]</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402510</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876</id>
	<title>Generators plus UPS FTMFW</title>
	<author>Anonymous</author>
	<datestamp>1268070540000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>2</modscore>
	<htmltext><p>Epic fail.</p><p>Any data center worth it's weight in dirt, must have UPS devices sufficient to power all servers plus all network and infrastructure equipment, as well as the HVAC systems too, for a minimum of at least 2 full hours on batteries, in case the backup generators have difficulty in getting started up and online.</p><p>Any data center without both adequate battery-UPS systems plus diesel (or natural gas or propane powered) generators is a rinky-dink, mickey-mouse amateur operation.</p></htmltext>
<tokenext>Epic fail.Any data center worth it 's weight in dirt , must have UPS devices sufficient to power all servers plus all network and infrastructure equipment , as well as the HVAC systems too , for a minimum of at least 2 full hours on batteries , in case the backup generators have difficulty in getting started up and online.Any data center without both adequate battery-UPS systems plus diesel ( or natural gas or propane powered ) generators is a rinky-dink , mickey-mouse amateur operation .</tokentext>
<sentencetext>Epic fail.Any data center worth it's weight in dirt, must have UPS devices sufficient to power all servers plus all network and infrastructure equipment, as well as the HVAC systems too, for a minimum of at least 2 full hours on batteries, in case the backup generators have difficulty in getting started up and online.Any data center without both adequate battery-UPS systems plus diesel (or natural gas or propane powered) generators is a rinky-dink, mickey-mouse amateur operation.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402510</id>
	<title>Re:the worst nightmare of data center peeps</title>
	<author>Anonymous</author>
	<datestamp>1268073360000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>Well, I&rsquo;m no expert, but it&rsquo;s not very hard to get a building water tight, now is it?</p></htmltext>
<tokenext>Well , I    m no expert , but it    s not very hard to get a building water tight , now is it ?</tokentext>
<sentencetext>Well, I’m no expert, but it’s not very hard to get a building water tight, now is it?</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31409878</id>
	<title>Re:the worst nightmare of data center peeps</title>
	<author>Eil</author>
	<datestamp>1268069640000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><blockquote><div><p>I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.</p></div></blockquote><p>Simple: the power equipment gets an unscheduled watering and your servers go down.</p><p>If you want to minimize the impact that a disaster can wreak on your servers in a datacenter, then you need to have your entire setup running and synchronously replicated in another datacenter.</p></div>
	</htmltext>
<tokenext>I 've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical ( including the dual redundant UPS systems ) in the data center.Simple : the power equipment gets an unscheduled watering and your servers go down.If you want to minimize the impact that a disaster can wreak on your servers in a datacenter , then you need to have your entire setup running and synchronously replicated in another datacenter .</tokentext>
<sentencetext>I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.Simple: the power equipment gets an unscheduled watering and your servers go down.If you want to minimize the impact that a disaster can wreak on your servers in a datacenter, then you need to have your entire setup running and synchronously replicated in another datacenter.
	</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820</id>
	<title>the worst nightmare of data center peeps</title>
	<author>Anonymous</author>
	<datestamp>1268070240000</datestamp>
	<modclass>Interestin</modclass>
	<modscore>3</modscore>
	<htmltext>i don't run a data center, but manage systems that rely on the data center 18 hrs/day 6 days/week. we pass upwards of $300m through my systems. I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.</htmltext>
<tokenext>i do n't run a data center , but manage systems that rely on the data center 18 hrs/day 6 days/week .
we pass upwards of $ 300m through my systems .
I 've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical ( including the dual redundant UPS systems ) in the data center .</tokentext>
<sentencetext>i don't run a data center, but manage systems that rely on the data center 18 hrs/day 6 days/week.
we pass upwards of $300m through my systems.
I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402204</id>
	<title>try employing the right people</title>
	<author>mjwalshe</author>
	<datestamp>1268072160000</datestamp>
	<modclass>Interestin</modclass>
	<modscore>2</modscore>
	<htmltext>try hiring some staff with telco experiance instead of kids with a perfect GPA scores from stanford and design the fraking thing better !</htmltext>
<tokenext>try hiring some staff with telco experiance instead of kids with a perfect GPA scores from stanford and design the fraking thing better !</tokentext>
<sentencetext>try hiring some staff with telco experiance instead of kids with a perfect GPA scores from stanford and design the fraking thing better !</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401768</id>
	<title>App Engine down again?</title>
	<author>bjourne</author>
	<datestamp>1268070000000</datestamp>
	<modclass>Insightful</modclass>
	<modscore>2</modscore>
	<htmltext><p>App Engine must be Googles absolutely most poorly run project. It has been suffering from outages almost weekly (the <a href="http://code.google.com/status/appengine/" title="google.com">status page</a> [google.com] doesn't tell the whole truth unfortunately), unexplainable performance degradations, data corruption (!!!), stale indexes and random weirdness for as long as it has been run. I am one of those who tried for a really long time to make it work, but had to give up despite it being Google and despite all the really cool technology in it. I pity the fool who pays money for that.</p><p>The engineers who work with it are really helpful and approachable both on mailing lists and irc, and the documentation is excellent. But it doesn't help when the infrastructure around it is so flaky.</p></htmltext>
<tokenext>App Engine must be Googles absolutely most poorly run project .
It has been suffering from outages almost weekly ( the status page [ google.com ] does n't tell the whole truth unfortunately ) , unexplainable performance degradations , data corruption ( ! ! !
) , stale indexes and random weirdness for as long as it has been run .
I am one of those who tried for a really long time to make it work , but had to give up despite it being Google and despite all the really cool technology in it .
I pity the fool who pays money for that.The engineers who work with it are really helpful and approachable both on mailing lists and irc , and the documentation is excellent .
But it does n't help when the infrastructure around it is so flaky .</tokentext>
<sentencetext>App Engine must be Googles absolutely most poorly run project.
It has been suffering from outages almost weekly (the status page [google.com] doesn't tell the whole truth unfortunately), unexplainable performance degradations, data corruption (!!!
), stale indexes and random weirdness for as long as it has been run.
I am one of those who tried for a really long time to make it work, but had to give up despite it being Google and despite all the really cool technology in it.
I pity the fool who pays money for that.The engineers who work with it are really helpful and approachable both on mailing lists and irc, and the documentation is excellent.
But it doesn't help when the infrastructure around it is so flaky.</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402262</id>
	<title>Re:what about having people onsite?</title>
	<author>Anonymous</author>
	<datestamp>1268072400000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>Google doesn't use traditional data centers.  They build theirs out of modules constructed from shipping containers. cf. <a href="http://blogoscoped.com/archive/2009-04-08-n39.html" title="blogoscoped.com" rel="nofollow">Google Data Center Video</a> [blogoscoped.com], <a href="http://www.engadget.com/2009/04/02/googles-data-center-secrets-revealed/" title="engadget.com" rel="nofollow">data center secrets revealed</a> [engadget.com].</p><p>So, remote monitoring, and then someone goes to check the module the alarm came from.  They may have to walk 100 meters to get to the module, though.</p></htmltext>
<tokenext>Google does n't use traditional data centers .
They build theirs out of modules constructed from shipping containers .
cf. Google Data Center Video [ blogoscoped.com ] , data center secrets revealed [ engadget.com ] .So , remote monitoring , and then someone goes to check the module the alarm came from .
They may have to walk 100 meters to get to the module , though .</tokentext>
<sentencetext>Google doesn't use traditional data centers.
They build theirs out of modules constructed from shipping containers.
cf. Google Data Center Video [blogoscoped.com], data center secrets revealed [engadget.com].So, remote monitoring, and then someone goes to check the module the alarm came from.
They may have to walk 100 meters to get to the module, though.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480</parent>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401712</id>
	<title>Don't they have</title>
	<author>Anonymous</author>
	<datestamp>1268069640000</datestamp>
	<modclass>None</modclass>
	<modscore>0</modscore>
	<htmltext><p>UPS's and backup generators? or some other onsite emergeancy power? (wind turbine, batteries, bunch of illegals on treadmills etc</p></htmltext>
<tokenext>UPS 's and backup generators ?
or some other onsite emergeancy power ?
( wind turbine , batteries , bunch of illegals on treadmills etc</tokentext>
<sentencetext>UPS's and backup generators?
or some other onsite emergeancy power?
(wind turbine, batteries, bunch of illegals on treadmills etc</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401576</id>
	<title>title should read "Google App Engine NOT a Cloud"</title>
	<author>Anonymous</author>
	<datestamp>1268068920000</datestamp>
	<modclass>Funny</modclass>
	<modscore>1</modscore>
	<htmltext><p>Obviously if the power goes out, and the service goes offline, then it WASN'T a cloud.  If it's a cloud, it can't go down.  If it goes down, it wasn't a cloud.</p><p>What's there to get?</p></htmltext>
<tokenext>Obviously if the power goes out , and the service goes offline , then it WAS N'T a cloud .
If it 's a cloud , it ca n't go down .
If it goes down , it was n't a cloud.What 's there to get ?</tokentext>
<sentencetext>Obviously if the power goes out, and the service goes offline, then it WASN'T a cloud.
If it's a cloud, it can't go down.
If it goes down, it wasn't a cloud.What's there to get?</sentencetext>
</comment>
<comment>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402728</id>
	<title>Re:Don't they have</title>
	<author>Glendale2x</author>
	<datestamp>1268074320000</datestamp>
	<modclass>None</modclass>
	<modscore>1</modscore>
	<htmltext><p>Actually no, Google doesn't use UPS systems if this is one of their designs that uses one small sealed lead acid battery per server.</p></htmltext>
<tokenext>Actually no , Google does n't use UPS systems if this is one of their designs that uses one small sealed lead acid battery per server .</tokentext>
<sentencetext>Actually no, Google doesn't use UPS systems if this is one of their designs that uses one small sealed lead acid battery per server.</sentencetext>
	<parent>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401712</parent>
</comment>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_8</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31411388
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402510
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_9</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31404786
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402172
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_15</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402022
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_19</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402656
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_2</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402788
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_6</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402262
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_13</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405394
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401700
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_17</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31409878
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_21</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403724
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402350
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_10</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403654
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401576
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_3</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402306
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_7</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31410828
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_14</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403272
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402510
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_0</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405096
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_20</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403530
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_18</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402278
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_1</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31420650
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402382
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401856
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_5</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402728
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401712
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_12</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31408764
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402172
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_11</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401998
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401576
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_4</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31453156
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574
</commentlist>
</thread>
<thread>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#thread_10_03_08_151248_16</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402516
http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820
</commentlist>
</thread>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.5</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402066
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.13</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401700
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405394
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.8</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401462
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.6</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401512
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.0</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401576
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401998
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403654
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.12</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401820
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402516
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402278
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402510
--http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403272
--http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31411388
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31409878
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.10</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401862
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.9</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403282
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.3</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401574
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402022
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31410828
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31453156
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403530
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.14</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403138
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.11</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401876
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402788
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402306
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402350
--http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31403724
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31405096
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402656
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.1</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401712
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402728
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.4</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401528
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.2</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401506
</commentlist>
</conversation>
<conversation>
	<id>http://www.semanticweb.org/ontologies/ConversationInstances.owl#conversation10_03_08_151248.7</id>
	<commentlist>http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401480
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402172
--http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31404786
--http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31408764
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31401856
--http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402382
---http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31420650
-http://www.semanticweb.org/ontologies/ConversationInstances.owl#comment10_03_08_151248.31402262
</commentlist>
</conversation>
