Wannabe er nå åpent for søknader for TG15! :)

Wannabe er nå åpent for søknader for TG15 og du kan lese her beskrivelsen Net:
http://wannabe.gathering.org/tg15/Crew/Description/Tech:Net

Om dette høres riktig ut for deg, så anbefaler jeg deg å registrere deg i wannabe og levere en søknad: http://wannabe.gathering.org/tg15/

Vi håper å se mange interessante søknader og søkere! :)

Posted in crew | Leave a comment

Internett og Fri Programvare i Norge – Lørdag kl 12:45

Hvem gjør noe for Internett og Fri Programvare i Norge?

Hør hva Nasjonalt Kompetansesenter for Fri Programvare (friprog.no) gjør for friprog i Norge, og hva Internet Society Norway Chapter (isoc.no) gjør for fremtiden til Internett.

De som stiller opp er
 – Christer Gundersen (friprog.no, første 20 min)
 – Salve J. Nilsen (isoc.no, siste 20 min)

Posted in the network effect | Leave a comment

Short status update…

Some of you may have experienced some problems with the internet, the wireless and the network in general. We have had some minor issues with the internet link, with the internal routing and the wireless. Everything was on track and working before 09 Wednesday morning, but we never really know how well things work before at least a few thousand participants actually arrive and connect to the network and put some load on it.

The wireless: 
We had some small problems with the servers to start with, and then some small problems with the configuration. The main problem here was that we had to prioritize the cabled network.

We are still working on improving the wireless solution and hope that we have everything optimized by tomorrow morning.

The internal network:
We don’t have one specific problem to point to, more like hundreds of small problems. The list is long and it contains everything from bug in software to missing parts and some human error. But there have not been any major incidents.

The internet, which is a two part problem:
1. We have 4x10Gig links in a port bundle down to Blix Solutions in Oslo. These were connected and tested OK on Friday. When participants arrived on Wednesday and the links became loaded with traffic we started to see problems with the load balancing. We removed two ports that weren’t performing well from the bundle and continued on 100% working 20Gig (2x10Gig).

This morning, around 11:00, SmartOptics arrived with new optical transceivers and converters. They checked the transceivers on the links we had problems with using an optical microscope and could see that they weren’t completely clean. Using special cleaning sauce, they managed to remove the dust and dirt from our transceivers, leaving it to us to put them back in the bundle, now in 100% working condition. Next year we’ll make sure to be more adamant about this before patching things together.

2. Origin, Steam, Blizzard, NRK, Microsoft, HP, Twitch… Some of these services rely on geolocation. There are multiple providers of geolocation service (like MaxMind), but the services usually charge money per database pull. This means that the cheaper the companies are, the longer between every pull. This means that we can be seen as being in Norway for some services that update often, but in Russia, Puerto Rico, Italy or Antarctica etc from companies that pull data from the geolocation database less frequently.

The reason for this is because our IP-address range is a temporary allocation from RIPE. RIPE has a pool with IP-addresses they lend out for a short amount of time to temporary events. This means that we are not guaranteed to get the same IP-addresses every year and that a lot of different events in different countries have been using the allocation in the months before us.

We are working continuously to solve this. We talk to Origin/EA and Valve, we try to NAT the most known and most used services through permanent Norwegian IP-addresses and we do ugly DNS-hacks. The sad fact is however that in the limited amount of time we have during TG, we won’t be able to solve this for every service.

Posted in status updates | Leave a comment

How to get optimus prime awesomesaucelicious wireless

5GHz… Connect to the broadcasted ESSID: “The Gathering 2014″ <- this one is only 5GHz and you are 100% surest to getest the bestest and freshestest frequencies. YAY!! ;)

Legacy clients with only 2,4GHz can connect to the “The Gathering 2014 2.4Ghz” ESSID.
2,4GHz is only best effort – the main focus is stable 5GHz

The password for both is: Transylvania
N.B. with capital T

Posted in Uncategorized | 2 Comments

We has intarwebz

Woot oO \:D/

Speedtest 3

TeleGW#sh int po 1
Port-channel1 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 001a.e316.a400 (bia 001a.e316.a400)
Description: Interwebz
Internet address is 185.12.59.2/30
MTU 1500 bytes, BW 40000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is unknown
input flow-control is on, output flow-control is off
Members in this channel: Te5/4 Te5/5 Te6/4 Te6/5

TeleGW#show ipv6 interfac po1
Port-channel1 is up, line protocol is up
IPv6 is enabled, link-local address is FE80::21A:E3FF:FE16:A400
No Virtual link-local address(es):
Description: Interwebz
Global unicast address(es):
2A02:ED01::2, subnet is 2A02:ED01::/64

Posted in blix, cisco, configuration, crew, eidsiva, status updates | 5 Comments

TG network since 1996

Asle, a guy who is very interested in networking and maybe more specific – the TG network, made this summary of the TG network (primary network vendor).

He have been crawling the internet up and down to locate as much accurate info about the core network as possible.

Check it out here: http://leknes.info/tgnett

Posted in fun | Leave a comment

The instability in the network at TG13

There’s been a lot of questions around the instability in the network at TG13. We are very sorry for not explaining this during TG13 or any time sooner then this. We are very, very sorry. But behold, the explanation is here… and I can say it in one short sentence:

We had a defective backplane on one of the two supervisor slots of the 6500 that terminated the internet connection.

First a short intro to Cisco, Catalyst 6500, sturdiness and reliability: the Cisco Catalyst 6500 is a modular switch that have been in production since 1999. It’s packed with functionality and is one of the most reliable modular switches as we know it.

You may ask, why did we end up with a defective one if they are so reliable… and you are right to ask. The answer is split in two… one: they have produced and sold many, many thousands of units – and even if they have very strict test routines there will always be some troublesome units. Two: this type of equipment is not meant to be shipped and moved back and forth as much as the Cisco Demo Depot equipment is. In the end the equipment will show problems caused by all the moving back and forth, not to mention the line cards always replaced.

We would have run the equipment in extensible testing, if we were a regular customer. And as a regular customer with a service agreement would we have contacted the Cisco TAC (Technical Assistance Center) and registered a service request. They would have either solved the problem remote if it were software bug or configuration error or RMA (return material authorization) the faulty hardware. This takes time, and that is something we usually calculate for when building a new, big network.

At TG we test all the equipment before we ship it to Vikingskipet. On the 6500 and 4500 we run the command “diagnostic bootup level complete”, and reboot the switches. When they are up again we have a complete diagnostic of all the line cards of the switch – and we can easily spot if something is wrong.  Both our 6500 last year passed the tests, so we shipped them up to Vikingskipet and configured them and everything was seemingly ok.

But not everything was ok. Not at all. We noticed something weird on Wednesday after the participants arrived. We had some strange latency and loss on the link between us and Blix Solutions. The interfaces did not have drops or errors, we just saw that more packets went out than return packets. We involved first Blix Solutions in the investigation, and they checked everything at their end and everything was ok there, so we involved Eidsiva bredbånd. They checked everything between Blix Solutions and us, and everything was fine also there.

This started to get stranger and stranger… Traffic was going out, but didn’t come back. No interface drops. No interface errors. Peculiar.

The setup was quite simple… 4x10Gig interfaces in a port-channel bundle towards Blix Solutions. 2x10Gig interface (X2) in each supervisor (Sup720).

So after some testing and some different hypothesis during Wednesday, we waited until the day was over and we went towards the Thursday morning and the traffic was dropped beneath 8Gig. At once it was stable beneath 8Gig, we forced all the traffic over to one and one 10Gig interfaces – first in the top most supervisor. The traffic was going smooth as silk, and I’m very sorry to say – but we did actually fail them over, one by one, to the 10Gig interfaces on the last supervisor for five minutes – and guess what? The internet traffic had drops and we have latency… We forced them back to the first supervisor and everything went smooth again.

We had an answer for why we had the problems, but then we had to find the solution to solve the problem on the fly, in production, before 6000 nerds woke up to life and started using internet traffic again! OMG! WHAT A PRESSURE! oO

As you may know are the transport between Vikingskipet and Blix Solutions “colored”, and we only had colored optical transceivers in the SFP+ format and SFP+ converter in X2 format. The rest of the 10Gig interfaces on the 6500 was XENPAK.

Luckily for us, Blix Solutions had put a TransPacket-box in each side, and this box can be configured with the right wavelengths (colors) on the interfaces. That gave us the possibility to translate the wavelength through the TransPacket-box and receive on a 10Gig XENPAK interface on regular 1310 single mode wavelength.

But this was not until a little bit out into the day, Thursday. So all instability you may have experienced on Wednesday and “early” Thursday was caused by this. After this, there should not have been any troubles or instabilities in the network as far as we know.

At the same time as we explain this we also have to applaud both Eidsiva bredbånd and Blix Solutions for their service. They were working with us every step of the way to find a solution. And we can really say we know now why Blix has the name they have… Blix Solutions – because they are very open and forward about finding a solution, however far fetched or unrealistic it is.

Thank you both Eidsiva bredbånd and Blix Solutions for the cooperation in TG13!

We really look forward to working with both of you for TG14 :)

Posted in blix, configuration, eidsiva | Leave a comment