Release engineering.

Hi.

I am a software engineer by day, and enjoy flight simulators by hobby, and this particular software family since Flanker times. I do admire the effort and workload put into DCS, and what many great things it allows. I know very well software is hard, living with a legacy codebase is harder, and that the domain of high-fidelity simulation is another step in complexity.

What doesn’t resonate with my professional point of view, is the reduced frequency and unpredictable cadence of releases, and their detrimental effect to perceived quality.

I do not claim to say whether it should be once every week or two weeks, but there is an agreement among software professionals that a regular delivery cycle positively impacts quality. How are "wrong branches" ending up in a release is beyond me. There is only one answer here - have a process, automate it, let people make decisions, let computers do the work.

It is also agreed that automated testing is beneficial to the state of software. You do have the “replay track” feature, as well as certain pre-set missions, I would expect it possible to automate a check, for example, if radar, using a certain location (given! known up front!), can see and lock a target. Similarly, if it can get rid of this target using a variety of ways. Let computers do the repeatable work, this is what they are for!

Bad releases do happen. But getting fundamental things broken repeatedly leads to questions. Getting them wrong over and might be funny to some.

Again, you have said old codebase, netcode etc. I get it. But these are within your control. Software engineering is not something invented 2 years ago. So is testing, continuous delivery, etc etc etc. What I would like to see, as a user, is a continuous, positive trend. I don’t care about a training bomb for the F18. I don’t care about SLAM, as much as I care about being able to first take off and land, and then get rid of some near air threats. I would like radar to work, and I would like basic weapon systems to be available.

From managing expectations point of view, this means: pause new features, fix things. Have a look at the release model Java adopted - whatever is ready, as a feature, for cutoff date, gets in, anything missed gets into the next one. What could keep happening is a feature release on a Monday, after tests by whoever does testing in the closed realms. That could lead to an easy go/no go decision, not at the last minute. Next patch, after that? Let’s say in one week, only bug fixes, those not critical. New features in two weeks? Sure. Tell us what your regular cadence is, and stick to it. Do we really need to keep asking “is there round the corner”?

Getting quality consistently up is IMHO more important than anything fancy for the Hornet. (and Viper deserves this 65D!).

This could lead to an improvement for when a hotfix is required, this can be done before the weekend, especially as many serious gaming organisations do run regular events towards the weekend.

Some issues, less core (old example: not being able to change audio in flight) can wait, a bit. I would, of course, like to see them fixed, too.

I can appreciate some areas (rendering engine, netcode) will take more time. They are doable, evolving software is hard, take some time. How to make things look better? Keep improving small things. “Small” like maps that match airports, default custom kneeboards per terrain, being able to delete some of existing kneeboard assets without affecting integrity checks, or things coming back. Data cartridges for airplanes? It doesn’t need to be a ground target evaporating feature to be “nice”. Restore lost RWR symbols? I am sure you have a list of things like that.

Such everyday activities should be quite simple to implement, but will make regular users feel slightly less annoyed (some way before “better”), and make regular use of the simulator a better experience.

The effects of the difficult few weeks are here. People I would usually fly with, fly with less. We have been forced to use airstarts, and land boat-based planes on airfields. With some, we are considering rolling back to 2.5.5 stable.

Please fix my sim.

Good points there. I am also a SW engineer and I have also been contemplating the state of the quality assurance in this project.

First, I suspect the breadth of the features in this project is huge. There are so many little details involved and there are also some scientifically complex features like flight modelling, radar operations and artificial intelligence. The scope alone means that one must use very light processes, and heavily prioritize tasks. I suspect that there just is not time to write automated testing for majority of code. After all, automated testing for a piece of software can double the effort plus the maintenance burden for the tests.

Second, automated testing is not well suited for all parts of software. For example, UI related functionality testing is much harder to test automatically. Many times one have to rely on human driven functional testing.Lately many issues have been UI related (lighting, VR rendering).

Even though I have pointed out some possible reasons for reduced automated testing, I strongly believe it should be used in places where it is applicable and beneficial. One such area could be performance regression testing, where certain performance level would be requirement for new feature to pass in to a release.

SE here. It’s painful as a customer and also painful to watch. I can’t add anything to the great posts above but us as users deserve more. We’re all pulling for you ED. Refine your processes if not for us, do it for your employees.

It it helps you understand further for you guys they are using SVN.......

+1 on this post.

But I think also Ed has to steel itself, because of the way the community has evolved and the way it as a company derives revenue... to not allow the community to dictate its release schedule to “stable” from “beta”...

The community also needs to address the ob vs stable dichotomy... a lot of it is using ob as if it were a stable release. This is fundamentally broken, and I’m not sure it’s a software problem.

Please @op don’t misunderstand me your post makes perfect sense...but in a world where frankly too many users use open beta and rely on open beta for their fun... as opposed to being work I’m just not sure how Ed balances this against those logical changes you propose.

Re @unclesam, definitely agree that automated testing can be a nil sum gain, so it is important to focus it on the right elements, it can be pretty hard to automate lighting tests, without doing some pretty complex machine learning models, so it’s definitely a horses for course sort of thing.

It it helps you understand further for you guys they are using SVN.......

Ugh, poor devs!

It it helps you understand further for you guys they are using SVN.......

I must be the only one who hates git with a burning passion, and would take SVN or TFS over it anyday of the week :P

Anyway the issue here isn't caused and won't be solved by any source control platform I would think :)

Open beta

I am sure stable build is working just fine, maybe ED should just get rid of open beta. Shit is ruining their image because ppl seems to refuse the difference between stable and beta.

Someone will come and say that stable has bugs too, then you see Windows with first version launched in 1985 still has several problems. God damn, I saw AIX bugs a lot of times. I've seen Linux bugs, Java and high-end equipments that costs millions of dollars simply freezing.

Don't come and say that because of agile mindset, or extensive tests bugs will stop, they will EVER exist. Softwares are written by people and even the automatization of bug testing are written by people.

We are not talking about a software for local market, this shit probably has millions and millions of code lines and we are talking about aerodynamics, complex math not a bunch of IFs, ELSEs.

I must be the only one who hates git with a burning passion, and would take SVN or TFS over it anyday of the week :P

 

Anyway the issue here isn't caused and won't be solved by any source control platform I would think :)

What, why?

Hg or Git, due to their design, are simply so much more productive and conductive to actual dev practices.

Branching in TFS is horrible, as it's a full blown deep copy. It's also gets painful for check-ins as team size gets bigger.

I'd never go back to TFS tbh.

There's clearly a source control problem though, maybe not directy the platform, but how it's managed, although the 2 are coupled. There really shouldn't be any instances of things "slipping in" to branches during merges. That said, I'd still urge ED to move to git/hg, considering they've got ~100 people on dev.

Open beta

I am sure stable build is working just fine, maybe ED should just get rid of open beta.

OPEN BETA is what is advertised by ED. They release their new money horses on it and stable gets rarely updated. For example the JF17 has only been added 2 weeks ago although its released since December. On top of that even in stable there are a shitton of old bugs and problems. Its just not as critical as "you cant play at all anymore".

Open Beta is a very good and usefull tool for the Devs and community if used right. The current Beta version is treated as an Dev Alpha. Critical and software breaking bugs get shipped to the customer. Basic functionality of the product is not guaranteed. That is ALPHA level of development not what a BETA is meant to be.

Getting back to OP:

Yes it would be very good to see them just merging features and updates as soon as they run well on OB to a stable version. I think the underlying issue with this is, that ED does not use a proper version control tool and they cant just update a few branches without using the whole thing. At least it seems to be the matter otherwise we would not see update after update where parts get left out or wrong versions being releases left and right.

Its a serious issue.

Open Beta is a very good and usefull tool for the Devs and community if used right. The current Beta version is treated as an Dev Alpha. Critical and software breaking bugs get shipped to the customer. Basic functionality of the product is not guaranteed. That is ALPHA level of development not what a BETA is meant to be.

Beta can brake, Release Candidate not.

Alpha is what you don't even want to release as it is just proof of concept.

The Open Beta is Open Beta, not even a Release Candidate.

But you shouldn't release something in beta that denies the using the software at all.

People have too high expectations from the Beta products.

The problem just is that in the closed testing team seems to be included with people who are not testers, but are content creators for YouTube. Those people shouldn't be there at all. Correct me if I am wrong in that, but those all should be experienced testers that are ready to do the testers tasks so that when something gets released to Open Beta, it is somewhat working.

+1 on this post.

 

But I think also Ed has to steel itself, because of the way the community has evolved and the way it as a company derives revenue... to not allow the community to dictate its release schedule to “stable” from “beta”...

 

The community also needs to address the ob vs stable dichotomy... a lot of it is using ob as if it were a stable release. This is fundamentally broken, and I’m not sure it’s a software problem.

^ this

we are now seeing the pitchforks come out and demand (untested) hotfixes in OB because some feature doesnt work, which wouldnt prevent further testing, but impacts game play, and everyone is playing rather than testing on OB. I play on open beta because everyone runs open beta and all servers run OB.

This really must change. Beta should be "safe" for developers to introduce new, risky, not properly tested changes or things that arent quite finished yes and may impact overall game play. And hotfixes should only be applied to bugs discovered in the stable release.

I dont know how to get there, maybe the beta should become closed, or in some other way users should be disincentivised from running OB as their primary/only game. Limit session times to 1 hour? Limit the number of users or server slots for OB ? force pink liveries? watermark over the screen? make an annoying popup that every 30 minutes demands the beta tester acknowledges he knows this is a beta release meant for testing rather than serious gaming.. ? whatever. But already we can all do our part by playing on the stable release and only do actual testing on OB.

Customers using OB for many reasons and I would not blame them:

1. All big MP servers are on OB basically beta testing DCS MP

2. "Stable" isn't really that stable, it is just OB a few patches before, and there are bugs older than me. Stable is also cramped with bugs and it will take even longer for them to get fixed in "Stable". Half of the SP missions and campaigns for example doesn't work on "stable" too, how is that stable ?

3. It is the same if you run Stable or have OB and when a shit patches like those are released just run update 2.5.6.xxxx and downgrade a few patches.

4. Lately the EA thing got more and more frequent and more and more Early. People want the content they paid $$ finished asap. If Modules are released in more finished state and stable was really more polished and stable then probably people would have stayed in stable. In fact, most of the Players and MP servers started to move to OB with the early access release of the Hornet and most of the big populated servers moved there so to use it.

EDIT: OB normally is for content which is considered almost ready for release and it requires mass testing from public to discover a corner cases bugs which beta testers could have missed. In case of DCS this is not the case, even stable is more beta than what used to be a beta branch.

Customers using OB for many reasons and I would not blame them:

 

1. All big MP servers are on OB basically beta testing DCS MP

 

2. "Stable" isn't really that stable, it is just OB a few patches before, and there are bug older than me. Stable is also cramped with bugs and it will take even longer for them to get fixed in "Stable". Half of the SP missions and campaigns for example doesn't work on "stable" too, how is that stable ?

This is largely a chicken and egg problem. Stable isnt stable at least in part because beta isnt treated as beta. Modules are sold in early access, because everyone has access to and most people run the OB. And servers run OB because most users run OB and users are then forced to run OB because all the good servers run OB.

It doesnt really matter who's to blame, but ED must find a way out of it.

This is largely a chicken and egg problem. Stable isnt stable at least in part because beta isnt treated as beta. Modules are sold in early access, because everyone has access to and most people run the OB. And servers run OB because most users run OB and users are then forced to run OB because all the good servers run OB.

 

It doesnt really matter who's to blame, but ED must find a way out of it.

Ofc it is on ED!

ED has releasing unfinished barely tested content to OB months before it goes to Stable and years before it can get finished. We all can guess what the reason for doing it.

Then they end up with a bunch of main features/modules which are massively used by the customers and in desperate need of fixes, then they release even more of this.

Now every one wants his already payed content finished and done asap. ED doesn't have resources to support and finish everything in time, so we end up in situation where they have to push not polished stuff to beta and push most of it to stable as well, otherwise stable would be lagging even more.

Instead, ED can release to beta finished and polished product keep in there to check for any unexpected bugs and move to stable in a week or so. This ofc means release less content but with better quality for the overall product.

Thats another problem that may compound this one; the lack of a steady income stream (ie, subscription fees) means ED can only feed its developers if they keep releasing new modules (regardless of their state of readiness) and they have every incentive to focus their developers time on new stuff rather than on fixing year old issues or even maintain the stable channel.These new early access module typically rely on users running OB, so EB kinda needs their customers to run OB.

That system is broken. But when a poll is published here if people want a subscription based model, >80% voted no. Something has to give.

Thats another problem that may compound this one; the lack of a steady income stream (ie, subscription fees) means ED can only feed its developers if they keep releasing new modules (regardless of their state of readiness) and they have every incentive to focus their developers time on new stuff rather than on fixing year old issues or even maintain the stable channel.These new early access module typically rely on users running OB, so EB kinda needs their customers to run OB.

 

That system is broken. But when a poll is published here if people want a subscription based model, >80% voted no. Something has to give.

Exactly fixing the core game does not generate money.

How would a subscription look like? It coule be a way out but im very hesitant. I spend almost around 800€ on DCS over the past 5 years. Thats a lot of money for a video game. I racing costs around 100€/year. So will i get 8 years of free subscription or would those 800€ be gone?

Id personally be really pissed if i had spend that much money and now need to pay to keep playing it. If they move to subscription they would also need to lower the module prices drastically. At least by 60-70%.

I spend almost around 800€ on DCS over the past 5 years. Thats a lot of money for a video game.

I just proposed a variant here:

https://forums.eagle.ru/showthread.php?t=277691

Basically subscription would be optional, provide early access (currently known as open beta) and access to all modules (or all ED modules) in both stable and beta.

But yes, the transition would need to be considered. Someone who already purchased all modules, should not be treated the same as a new player who hasnt spent a dime yet. I think solutions can be found for that, for instance, every module you already own would give you, say, 6 months of free unlimited access. After that, you either pay like everyone else. Or you dont as in my proposal, but then you lose early access and access to modules you dont own.

Though it was said many times now. Once again. Build with critical bugs, seen from the first glance can't be called BETA. It is just developers bad work to give such build for testing, and shame for managers to roll this to public. If it is unusable what shall we test?

Exactly fixing the core game does not generate money.

 

 

How would a subscription look like? It coule be a way out but im very hesitant. I spend almost around 800€ on DCS over the past 5 years. Thats a lot of money for a video game. I racing costs around 100€/year. So will i get 8 years of free subscription or would those 800€ be gone?

Id personally be really pissed if i had spend that much money and now need to pay to keep playing it. If they move to subscription they would also need to lower the module prices drastically. At least by 60-70%.

Sent from my Redmi 4 using Tapatalk

I was perfectly fine with the old “stable version” ran great. I had OB on my system in case I wanted to actually play with somebody instead of just Ai.

Due to external pressures or whatever, they just dumped the OB issue into “stable “ and screwed up both branches. Not sure I’ll leave, but certainly am finding other ways to kill time. Frankly tired of spending the first 30 min reconfiguring stuff only to get “dumped for an “idle connection” or pull in behind a bandit and not be able to get a missile off the rail. Oh, I also noticed that the F-14 starts on deck kneeling again...... really?

Having worked in corporate IT for over 30 years and involved in ITIL process implementation and also having worked on some games as a tester over the last couple of decades my conclusion is there is no one size fits all in terms of process, far from it.

Even in the same organisation, the process for managing software in mission critical production facilities is massively different to say internal media communications. Stuff you might get fired for in the former is encouraged as being "agile" in the latter.

In my experience the games industry is very much at the agile end of things. IMHO probably to the point of being cavalier to a risk adverse IT manager in a corporate world!

The scenes seem to be repeated on every games forum, new release or update and stuff gets broken or new bugs emerge and the boards go crazy. Coming from a corporate background with control over standardised computer builds I have a little sympathy but the time between freezing changes and release to me seems a little insane (not talking DCS here, just all games in general).

However, the fact it is common practices doesn't make it ok, it just says to me this is a necessary evil in the business and applying corporate IT experience to the games industry isn't really applicable.

My gut feel is that in spite of the noise on the forums, getting stuff out there and fixing it after is preferable to long delays getting everything tested to the nth degree for this industry, not mine or whatever industry people commenting may be working in. The comments like "I would get fired in my job if I did this" may hold true, but this isn't the same job.

I must be the only one who hates git with a burning passion, and would take SVN or TFS over it anyday of the week :P

 

Anyway the issue here isn't caused and won't be solved by any source control platform I would think :)

No, it’s just a tool like anything else. Their process is flawed and will continue to be if they keep implementing it they way that they have.