Operation "Blue Flag" - 24/7 PvP Campaign - ROUND 9

Well from what someone has written here he solved it with VPN as the VPN is not going via grnet.gr (which shows the 77.8 % packet loss) so it helped some...

But clearly VPN is a temporary solution if the root cause is the server hardware itself...

But clearly VPN is a temporary solution if the root cause is the server hardware itself...

server hardware of grnet, not buddyspike (afaik), so we can only change location tempoary and hope grnet still gets some money of IWF/EZB... (and uses it to upgrade the hardware instead of paying back debts!!!)

But clearly VPN is a temporary solution if the root cause is the server hardware itself...

VPN cannot be a solution to a server harware issue, it may be a solution to network congestion issue.

I would advise to not mix 2 issues and 2 fixes at the same time since you won't know what fixes what.

Fix the network issue first, then go for fixing the hardware issue, if (some of) the problem remains

Whisper. We can't fix a hardware issue of a company. We can only bypass that company network...

Whisper. We can't fix a hardware issue of a company. We can only bypass that company network...

Errrr ... That's not what I was saying :)

I think OperatorJack is talking about BS server hardware issues which would cause some of the disconnect problems, not the potential routing issue which would cause some other of the disconnect problems. That's why I talk about 2 different issues and 2 different fixes.

Well from what someone has written here he solved it with VPN as the VPN is not going via grnet.gr (which shows the 77.8 % packet loss) so it helped some...

Yes but you see, its dropping the ICMP ping packets when it is entering the box to be responded to. There is a huge diffrence between forwarding the packet and responding to it, its handled by 2 complete separate chips.

The loss you see has NOTHING to do with the traffic going to the server. UDP and TCP can go just fine trough the box but when the box itself responds to the ping it shits the bed. A traceroute is simply a ping with an incrementing hop counter, thats how you can see the trace because when the hop counter reaches 0 the router will not route the packet any more and will return the ping. There are no packetloss on this connection, the second picture proves it.

When was your traceroute done, Cisco?

I'm basing my assumptions on what is found on post #49, by microvax : https://forums.eagle.ru/showpost.php?p=2923442&postcount=49

On this MTR, all hops before the Cogent-Geant link show zero packet drop, and all hops from the Cogent-Geant link and after show packet drops

While I agree that single hop showing packet drop are completely unconclusive due to the many ICMP throttling in place, the probability that all hops before one point never throttle ICMP while all after said point do throttle ICMP is really, really low.

The only possible throttling explanation would be that all hops showing drops are managed by the same entiry, thus may be the same hardware with the same configuration.

That's not our case, as at least 2 orgs are routing after the Cogent/Geant link, GEANT, the entity aggregating europeans university and research networks, and GR-Net, the Greek university network. Seeing the varietty of networks aggregated by Geant (like RENATER that I know a lil bit :) ), I'm confident Geant and GR-Net are pretty independant from each others.

 

So based on this traceroute, I'm logically going for the conclusion that the Cogent-Geant links really looks faulty.

 

At what time was your MTR done? I've tested as well last week, going through Cogent-Giant link, didn't see the PL listed in microvax post. I tested at 11AM. I assumed microvax tested at european "Internet business hour" (18:00 - 00:00) where traffic is way higher than the rest of the day.

It was tested at 11:00-11:30 GMT+1, i agree not the best time to test but i will do more tests later today.

Im 100% certain that the problem is not network related, i am after all a Network Engineer and i do that for a living.

But clearly VPN is a temporary solution if the root cause is the server hardware itself...

VPN is not a solution at all because the problem is not related to the network, i cant speak for the hardware side of things but its deff not the network.

Listen, the ICMP is forwarding just fine, NONE of the cogent equipment is dropping packets so stop blaming cogent, the address "geant.demarc.cogentco.com" means that the IP address is terminated in customer eqipment (where cogent is the transit provider and therefor supplying the IPs on the linknet for the BGP session).

 

There are no loss on ICMP on other than the one hop everyone is reporting loss on, for the last time: This is not packetloss, the only reason this hop is dropping ICMP is because the hardware running there is old or overworked, this is completely normal for network equipment older than 10 years.

 

I did the test using Hibernia networks ring server, Hibernia is a level 2 provider and buys transit from Cogent, the routes for the DCS server goes completely over cogent as you can see.

 

When looking at an mtr you have to see at the loss after the initial loss, as you can see there is no loss after pop 17. and there is 0% loss on the box itself, this means there are no packet loss.

 

 

If we ignore the traceroute for a while we also see that there is no packetloss to the server itself (even tho this is shown in the traceroute above i feel its better to simplify it with this picture)

 

 

This number here, its all you have to care about, a traceroute will show you where there is congestion or the problem is down the line but the ping as you see here is the absolute truth. There it is, clear as crystal that there is 0% packetloss to the server over cogent.

 

For the sake of argument i have included the stats from my server in Norway.

 

 

The conclusion is that the addition of a VPN will only make it worse (because of MTU issues) and any perceived benefit gained by using is nothing but placebo.

 

This test was taken at roughly 11:00-11:30 GMT+1 2016-10-18, i will repeat this test at 21:00 GMT+1 which is where the internet is at its peak in terms of bandwidth to rule out any suspected link congestion.

If I had the same results as you, I would make the same conclusions, but here look at my traceroutes:

At peak time the PL got to something like 25%. In the morning [everything before 12] it usually is between 0 and 5%. I dunno how pl looks during peak time ATM.

I am not 100% sure that the simultaneous mass disconnects are network related, I am pretty sure its not only network related. But the Problem I had in the beginning and a few other people as well, like singular disconnects without even beeing able to taxi or choose a slot before timeout were pretty surely connected to the Pl right there. Because independant of buddyspike server client count it was correlated to overall PL to buddyspike server.

The hardest part will be to find the right POC to blame.

Just unless you actually pay for the line you have basically no lever to get something done, that's the bad apart.

Good thing is, if 35k others have files a ticket...you are not alone and shows the size of problem :)

It was tested at 11:00-11:30 GMT+1, i agree not the best time to test but i will do more tests later today.

Im 100% certain that the problem is not network related, i am after all a Network Engineer and i do that for a living.

Sorry but that's not an argument :) (what do you think I do for a living? ;) )

I still think Microvax packet loss distribution along his path is pointing toward a network congestion more than any other explanation so far. And your argument didn't tell me either why you are 100% sure, since you don't seem to take microvax packet loss distribution along traceroute into account. The fact same some router could throttle some ICMP packet doesn't mean each time you see packet loss, it's a router throttling ICMP.

Another point, we only see the Internet -> BS server in these MRTs. Perhaps the issue lies in the other direction, if the return path toward microvax differs from yours that may explain the differences.

I tried to ping with Record Route option activated, but that was (as expected) ignored by routers on the way :(

Errrr ... That's not what I was saying :)

I think OperatorJack is talking about BS server hardware issues which would cause some of the disconnect problems, not the potential routing issue which would cause some other of the disconnect problems. That's why I talk about 2 different issues and 2 different fixes.

server hardware? But Greg hasn't even stated (yet) that there are performance issues on the BS server hardware...

Sorry but that's not an argument :) (what do you think I do for a living? ;) )

I still think Microvax packet loss distribution along his path is pointing toward a network congestion more than any other explanation so far. And your argument didn't tell me either why you are 100% sure, since you don't seem to take microvax packet loss distribution along traceroute into account. The fact same some router could throttle some ICMP packet doesn't mean each time you see packet loss, it's a router throttling ICMP.

Another point, we only see the Internet -> BS server in these MRTs. Perhaps the issue lies in the other direction, if the return path toward microvax differs from yours that may explain the differences.

I tried to ping with Record Route option activated, but that was (as expected) ignored by routers on the way :(

Yeah its a pity that record route mostly gets ignored. :/

Sorry but that's not an argument :) (what do you think I do for a living? ;) )

I still think Microvax packet loss distribution along his path is pointing toward a network congestion more than any other explanation so far. And your argument didn't tell me either why you are 100% sure, since you don't seem to take microvax packet loss distribution along traceroute into account. The fact same some router could throttle some ICMP packet doesn't mean each time you see packet loss, it's a router throttling ICMP.

Another point, we only see the Internet -> BS server in these MRTs. Perhaps the issue lies in the other direction, if the return path toward microvax differs from yours that may explain the differences.

I tried to ping with Record Route option activated, but that was (as expected) ignored by routers on the way :(

There is no congestion between grnet and cogent and there never have been, without knowing microvax'es IP i cannot say what route the traffic takes on the way out.

https://mon.grnet.gr/rg/281645/details/#tabs=overview

Access port to that the server uses. No congestion.

https://mon.grnet.gr/rg/282069/details/

GRNET to GEANT cogent, 20gbit link 10% used, no congestion (MPLS? Peering or partial transit?)

Having hard time finding graphs between Cogent and GEANT but i think the chance of it being congested between a Tier 1 and a Tier 2 is VERY VERY low.

There is no network issues

There is no congestion between grnet and cogent and there never have been, without knowing microvax'es IP i cannot say what route the traffic takes on the way out.

 

https://mon.grnet.gr/rg/281645/details/#tabs=overview

Access port to that the server uses. No congestion.

 

https://mon.grnet.gr/rg/282069/details/

GRNET to GEANT cogent, 20gbit link 10% used, no congestion (MPLS? Peering or partial transit?)

 

Having hard time finding graphs between Cogent and GEANT but i think the chance of it being congested between a Tier 1 and a Tier 2 is VERY VERY low.

 

There is no network issues

I just posted fresh MTRs.

There is no congestion between grnet and cogent and there never have been, without knowing microvax'es IP i cannot say what route the traffic takes on the way out.

 

https://mon.grnet.gr/rg/281645/details/#tabs=overview

Access port to that the server uses. No congestion.

 

https://mon.grnet.gr/rg/282069/details/

GRNET to GEANT cogent, 20gbit link 10% used, no congestion (MPLS? Peering or partial transit?)

The GRNET connections were not the ones incriminated.

GR-Net to GEANT links are "upstream" links, GEANT acts as an aggregation european network for national universities networks. Gr-Net has access to Internet through : 1) Geant network, 2) direct peerings in IX in Greece (GRIX), with notable Internet actors (GAFA and such) present there. It's all described in RIPE database.

GEANT has 2 Tier 1 transit providers, Cogent and Level3, + a whole lot of peerings everywhere as expected from a network this size in Europe. Same, all described in RIPE.

Having hard time finding graphs between Cogent and GEANT but i think the chance of it being congested between a Tier 1 and a Tier 2 is VERY VERY low.

That said, the return path may indicate a network issue somewhere else than the Cogent / Geant link.

But basing the "it's not a network issue" conclusion on the fact it's a Tier 1 network (Cogent) isn't wise, if you ask me, I've had many issues of this type in the past with Tier1 peering links (particular Tier1 being .... Cogent, ending us into a "never Cogent as transit provider again" policy. That was pre-MegaUpload shutdown, so things are perhaps better today).

There is no network issues

We tried.

Every field spawns two F-16's. These are taken care of easily, but they take multiple missiles each due to AI knowing everything, etc.

To exclude the rise of bots F16, it is necessary to damage the runway missile Kh-25ML and Kh-29.

How can one man be so wrong.

Wanting a period correct model doesn't make us graphics whorls.

The Mig-29S is period correct too. If you want A's up close that's fine too, but I don't really see the need. It's just going to be so much work for Greg to swap them in and out as he allows 120/77 depending on round etc...

The conclusion is that the addition of a VPN will only make it worse (because of MTU issues) and any perceived benefit gained by using is nothing but placebo.

Read into it however you want. Reality shows since using the VPN to connect I haven't been disconnected once. When I play it's usually for an entire 3 hour session, so while I was happily connected for 3 hours last night attacking VA in my Ka-50 I was watching the mass disconnects from all the poor souls who don't have the luxury of using a VPN.

But clearly VPN is a temporary solution if the root cause is the server hardware itself...

That's what I'm trying to say. The server isn't the problem because when using the VPN to get around the congested networks I have zero problems with the server.

Blue flag Teamspeak is down.

Got an official report of congestion issues between Cogent and Deusche Telekom (Microvax provider) right now, we are facing issues with customers here in France because of this.

I'll get a hand on the Cogent official statement about this, but that explains Microvax traceroute and VPN workaround being efficient for some people.

EDIT : here it is, Cogent response from Oct 5th to our ticket, still unresolved :

our peerings with DTAG face saturation during main business hours.

 

Cogent and DTAG exchange traffic under our settlement free peering agreement.

 

Cogent has asked DTAG to set up more peering connections in Europe but unfortunately DTAG has not decide yet how to proceed.

 

 

 

We have repeatedly requested augments to these congestion points and hope DTAG will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact DTAG customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved

Usual peering issues where both parties don't want to pay for the little investment needed for upgrade. Like I said, not the first time we see this from Cogent, and why I have a rather bad history with them as a provider.

That means german players are probably gonna face issues on BS server.

Got an official report of congestion issues between Cogent and Deusche Telekom (Microvax provider) right now, we are facing issues with customers here in France because of this.

I'll get a hand on the Cogent official statement about this, but that explains Microvax traceroute and VPN workaround being efficient for some people.

 

EDIT : here it is, Cogent response from Oct 5th to our ticket, still unresolved :

DTAG is Deutsche Telekom AG.

Usual peering issues where both parties don't want to pay for the little investment needed for upgrade. Like I said, not the first time we see this from Cogent, and why I have a rather bad history with them as a provider.

 

That means german players are probably gonna face issues on BS server.

Unfortunatly...

ok, so i have to change my internet provider. I hope this will be done before Round 9 is ended

Wow, I can see now why one of our German buddy get disconnect many many times...

At this point, I feel sorry for Red. They need lots of players to do a coordinated attack on Tblisi, but d/c's are killing them very effectively.

That means german players are probably gonna face issues on BS server.

Afaik USA players also had this issue...

ok, so i have to change my internet provider. I hope this will be done before Round 9 is ended

You can't be serious!!! xD (or are you??? o_O)

I know that it's an additional load for the Server and Clients, but war ca Not be ein without Boots on the ground. I don't mean the troops and mortars that are placed by Helis...they are static...i mean soemthing like a moving Tank group that has to be protected and Drives to airfields and cities to take them. What do you think about that?

Got an official report of congestion issues between Cogent and Deusche Telekom (Microvax provider) right now, we are facing issues with customers here in France because of this.

I'll get a hand on the Cogent official statement about this, but that explains Microvax traceroute and VPN workaround being efficient for some people.

 

EDIT : here it is, Cogent response from Oct 5th to our ticket, still unresolved :

DTAG is Deutsche Telekom AG.

Usual peering issues where both parties don't want to pay for the little investment needed for upgrade. Like I said, not the first time we see this from Cogent, and why I have a rather bad history with them as a provider.

 

That means german players are probably gonna face issues on BS server.

Yup typical DTAG.

Gonna launch dem complaint/ticket ICBMs at their support outlets ! ;D