Testng multiple suites with different listeners

I am using testng, having 3 kinds of tests:

1. without listener

2. listener A

3. listener B

Now how do i make testng.xml that assigns said listeners to the tests? I have tried using following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">


<suite name="allSuites">
    <suite-files>
        <suite-file path="testA.xml"/>
        <suite-file path="testB.xml"/>
    </suite-files>
</suite>

testA.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="First Suite">
    <listeners>
        <listener class-name="com.listeners.ListenerA"/>
    </listeners>
    <test name="Test A tests">
        <classes>
            <class name="com.tests.TestA"></class>
        </classes>
    </test>
</suite>

testB.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Second Suite">
    <listeners>
        <listener class-name="com.listeners.ListenerB"/>
    </listeners>
    <test name="Test B Tests">
        <classes>
            <class name="com.tests.TestB"></class>
        </classes>
    </test>
</suite>

but this doesnt work. The problem is that other tests without suite are being executed with one of those listeners. How do i tell testng not to use any listeners to any other tests?

THanks for help!