Nowadays, providers of wireless services try to find appropriate ways to increase user data throughput mainly for future 5G cellular networks. Utilizing the unlicensed spectrum (ISM bands) for such purpose is a promising solution: unlicensed frequency bands can be used as a complementary data pipeline for UMTS LTE (Universal Mobile Telecommunication System - Long Term Evolution) and its advanced version LTE-Advanced, especially in pico- or femtocells. However, coexisting LTE and WLAN services in shared ISM bands at the same time can suffer unwanted performance degradation. This paper focuses predominantly on co-channel coexistence issues (worst case) between LTE and WLAN (IEEE 802.11n) services in the ISM band. From the viewpoint of novelty, the main outcomes of this article are follows. Firstly, an appropriate signal processing approach for coexisting signals with different features in the baseband is proposed. It is applied in advanced link-layer simulators and its correctness is verified by various simulations. Secondly, the influence of IEEE 802.11n on LTE data and control channel performance is explored. Performance evaluation is based on error rate curves, depending on Signal-to-Interference ratio (SIR). Presented results allow for better understanding the influence of IEEE 802.11n on the LTE downlink physical control channels (PCCH) and are valuable for mobile infrastructure vendors and operators to optimize system parameters.