Lessons learend (from failuere) while I developing a log server for Windows

Wrote this post in Jan 2015 as a quick draft, and remained as it is still today. I thought it’s not that useful so tried to delete it, but changed mind and released here. Hope it helps.

Recently I developed a logging server based on C#/asp.net on Windows 2008 R2 server. The server primarily does receiving log files from clients and write into database and also drop files to that server. After I released it to production, I met some unexpected issues.

1. Huge(99.9%) memory consumption issue

The logging server writes huge excessive IO to the disk and it instantly caused the os SYSTEM process to consume almost all physical memories. I searched online and analyzed it through the Sysinternals’ RAMMap. It turns out the OS’s system cache takes all the memories. You can find out details from following links:

http://www.microsoft.com/en-us/download/details.aspx?id=9258 Windows Dynamic Cache Service

Sysinternals http://technet.microsoft.com/en-us/sysinternals/bb897561

KB article http://support.microsoft.com/kb/976618/

[Korean] http://snoopybox.co.kr/1635

Lesson learned: Reduce the file IO as much as you can!





2. MongoDB

  • Don’t forget to use the indexing
  • You can figure it out whether it uses the index or not by using ‘Explain’


3. Deleting files from disk will take longer than you expect

  • My initial design approach was complete wrong and failure. The server used to drop huge number of files into the physical disk, and just deleting the very small portion of files required actually even hours.

Lesson learned: Does your server collect a number of files? Store it into the database directly.



Running Python Flask on a local IIS (not Azure!) with wfastcgi.py

There are already a lot of resources dealing with how to setup Django with IIS, or configure IIS to support WSGI with various tools such as Helicon Zoo, isapi-wsgi, and FastCGI.

FastCGI looked good among others because I wanted to go with least 3rd party stack and the Microsoft distribute the wfastcgi.py through PTVS. However, there’s a still one problem. I couldn’t find the document explains how to set up a local IIS to work with FastCGI and Flask.

Thanks to PTVS, it’s already well implemented and provided. The PTVS contains the wfastcgi.py, and web.config which set required environments to run. Here’s more detailed step:

1. Install wfastcgi.py. The PTVS official homepage has the link to download wfastcgi.py MSI installer.



The wfastcgi MSI installer creates local copies of wfastcgi.py to python installed directory as below:


2. Install FastCGI to IIS and add wfastcgi.py handler to it. The official PTVS page has the required command to ease the steps here:


start /wait %windir%\System32\\PkgMgr.exe /iu:IIS-CGI

%windir%\system32\inetsrv\appcmd set config /section:system.webServer/fastCGI “/+[fullPath=’c:\Python27\python.exe’, arguments=’C:\Python27\Scripts\wfastcgi.py’]”

The wfastcgi.py MSI installer also seems to add the handlers above automatically if FastCGI is already configured, but it’s not sure. You can now see the FastCGI listed installed Python paths.


3. Now it’s time to edit the web.config for Flask/IIS. If you created the Flask project with PTVS addon, it already contains the required web.config to support deploy to Azure. However, in order to run it on local IIS, it needs a little modification. Let’s see the original web.config first PTVS generated.

<?xml version=”1.0″?>
<!– Generated web.config for Microsoft Azure. Remove this comment to prevent
modifications being overwritten when publishing the project.
<add type=”Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35″ name=”AzureDiagnostics”>
<filter type=”” />
<add key=”WSGI_ALT_VIRTUALENV_HANDLER” value=”FlaskWebProject1.app” />
<add key=”WSGI_ALT_VIRTUALENV_ACTIVATE_THIS” value=”%ROOTDIR%\env\Scripts\activate_this.py” />
<add key=”WSGI_HANDLER” value=”ptvs_virtualenv_proxy.get_virtualenv_handler()” />
<add key=”PYTHONPATH” value=”%ROOTDIR%” />
<compilation debug=”true” targetFramework=”4.0″ />
<modules runAllManagedModulesForAllRequests=”true” />
<add name=”Python FastCGI” path=”handler.fcgi” verb=”*” modules=”FastCgiModule” scriptProcessor=”%INTERPRETERPATH%|%WFASTCGIPATH%” resourceType=”Unspecified” requireAccess=”Script” />
<rule name=”Static Files” stopProcessing=”true”>
<match url=”^/static/.*” ignoreCase=”true” />
<action type=”Rewrite” url=”^/FlaskWebProject1/static/.*” appendQueryString=”true” />
<rule name=”Configure Python” stopProcessing=”true”>
<match url=”(.*)” ignoreCase=”false” />
<action type=”Rewrite” url=”handler.fcgi/{R:1}” appendQueryString=”true” />

Few things. First, I removed the <system.diagnostics> section which installs Azure diags listener. Second, the IIS cannot understand the %ROOTDIR% variable so you should convert it all absolute path for that server you about to run.

So it’ll look like this:

<add key=”WSGI_ALT_VIRTUALENV_HANDLER” value=”FlaskWebProject1.app” />
<add key=”WSGI_ALT_VIRTUALENV_ACTIVATE_THIS” value=”D:\repo\web\FlaskWebProject1\FlaskWebProject1\env\Scripts\activate_this.py” />
<add key=”WSGI_HANDLER” value=”ptvs_virtualenv_proxy.get_virtualenv_handler()” />
<add key=”PYTHONPATH” value=”D:\repo\web\FlaskWebProject1\FlaskWebProject1” />

The generated web.config has a python handler mapped, but the path also needs to be updated to point the direct wfastcgi.py path as follows:

<add name=”Python FastCGI” path=”handler.fcgi” verb=”*” modules=”FastCgiModule” scriptProcessor=”C:\Python34\python.exe|C:\Python34\Scripts\wfastcgi.py” resourceType=”Unspecified” requireAccess=”Script” />

It’s enough to edit a web.config. However, if you still encounter a strange 500 internal error, please make sure the local IIS has the URL Rewrite 2.0 module installed. In my case, whenever I tried to open the edited web.config from IIS manager, it gives me that “There was an error while performing this operation. … web.config” error so that I couldn’t open a web.config nor edit mapping handlers. Fortunately, I found those answers from SO and solved it by installing URL Rewrite 2.0 through the ‘Web Platform Installer’. Microsoft Download site also provides a standalone exe.




Done. If the IIS successfully loaded web.config, then it’ll show the correctly mapped Python wfastcgi handler at ‘Handler Mappings’ as follows:

handler1 handler2

If you have any questions, please leave a comment. Thanks for reading.

Extending Python with Visual C++ and PTVS

Last month there was a chance I gave a talk explaining how to start the PTVS(Python Tools for Visual Studio) through the MS virtual academy. The talk was live recorded and currently is being shared following URL:


It’s Korean though. Throughout the demo, except the very basic examples I also included demos showing how to build a Python extension with C++ which is integrated greatly with Visual Studio C++ compiler and the PTVS, and how to debug cross platform python modules running on Raspberry Pi.

Although the C++ python extension example I wrote was super simple and copy&pasted almost from python.org example, I uploaded mine here including C++ and python sample sources because it wasn’t easy to find as complete visual studio solution form.

GitHub repo URL: https://github.com/heejune/demo-python-PyCExtDemo1

Also if you want to start building a C++ python extension with Visual Studio and PTVS please refer to following steps:

1. Specify the include directory contains python headers.


2. Add python27.lib or appropriate version corresponds to.


3. Add libs directory into additional lib path.


4. Lastly, don’t forget enabling ‘Mixed mode debugging’ and matching python environment’s architecture version with python extension architecture (as follows)