Lessons learend (from failuere) while I developing a log server for Windows

Wrote this post in Jan 2015 as a quick draft, and remained as it is still today. I thought it’s not that useful so tried to delete it, but changed mind and released here. Hope it helps.

Recently I developed a logging server based on C#/asp.net on Windows 2008 R2 server. The server primarily does receiving log files from clients and write into database and also drop files to that server. After I released it to production, I met some unexpected issues.

1. Huge(99.9%) memory consumption issue

The logging server writes huge excessive IO to the disk and it instantly caused the os SYSTEM process to consume almost all physical memories. I searched online and analyzed it through the Sysinternals’ RAMMap. It turns out the OS’s system cache takes all the memories. You can find out details from following links:

http://www.microsoft.com/en-us/download/details.aspx?id=9258 Windows Dynamic Cache Service

Sysinternals http://technet.microsoft.com/en-us/sysinternals/bb897561

KB article http://support.microsoft.com/kb/976618/

[Korean] http://snoopybox.co.kr/1635

Lesson learned: Reduce the file IO as much as you can!

 

http://blogs.msdn.com/b/ntdebugging/archive/2009/02/06/microsoft-windows-dynamic-cache-service.aspx

http://blogs.msdn.com/b/ntdebugging/archive/2007/11/27/too-much-cache.aspx

 

2. MongoDB

  • Don’t forget to use the indexing
  • You can figure it out whether it uses the index or not by using ‘Explain’

 

3. Deleting files from disk will take longer than you expect

  • My initial design approach was complete wrong and failure. The server used to drop huge number of files into the physical disk, and just deleting the very small portion of files required actually even hours.

Lesson learned: Does your server collect a number of files? Store it into the database directly.

Thanks,

Heejune

Beginning the coroutine with Visual Studio 2015 Update 3 Part 2

As we discussed through Part I, co_wait keyword requires sort of “resumable thing”. So what if we want to use co_await with time delta?

For example, you can see the code which the co_await takes std::chrono::duration in order to suspend for the specified time from here: https://github.com/Microsoft/cppwinrt/blob/master/10.0.14393.0/Samples/JustCoroutines/Main.cpp#L31

The answer is the fact that vc++ compiler allows co_await to be overloaded. So, we can overload the co_await keyword and returns the “resumable thing” within the overloading implementation.

See the code here. https://github.com/Microsoft/cppwinrt/blob/master/10.0.14393.0/winrt/base.h#L8897

If you follow the resume_after(), then you’ll clearly see it actually returns the instance of resumable thing struct which contain implementations of await_ready, await_suspend, and await_resume.

https://github.com/Microsoft/cppwinrt/blob/master/10.0.14393.0/winrt/base.h#L8643

Let’s test with another test sample here. https://github.com/heejune/wrl-cppwinrt-sample/blob/async-support/SampleLib.Shared/DemoCore.cpp#L81

In this sample, I called the winrt’s co_await as the above sample illustrated.

And if I step into the co_await… we hit the overloaded co_await.

part_ii_1

Step in again, then finally it creates resume_after struct instance and calls the await_suspend.

part_ii_2

part_ii_3

It spawns timer thread with specified time, and finally resume_after::callback static callback is being called.

part_ii_3-4png

The callback will finally call resume().

part_ii_4

And the resume() transfers its context into where the co_await being called.

part_ii_5

And lastly, the coroutine function hits the co_return keyword after looping 5 times,

part_ii_6

And reached the end.

Next time, I’d like to talk about another co_await adapter ‘await_adapter’ which makes winrt IAsync.. types possible to be awaitable. Thanks.

Heejune