Error when using a temp table and join with a physical table

Well i have never seen this exception before,  a good day to learn something new:



select * from dbo.[a]

create table tempdb.#b (temp_number varchar(11))

insert into tempdb.#b (temp_number)
select dnb_number from openquery ([XYZ], 'select dnb_duns_number from dnb_store_data')

create nonclustered index temp_number_idx on tempdb.[#b] ([temp_number])

select * from tempdb.[#b] a (nolock)
where not exists (
select [matched number] from dbo.[a] b (nolock)
where a.[temp_number] = b.[matched number])


(381618 row(s) affected)
Exception has been thrown by the target of an invocation.

(1 row(s) affected)

On searching found out it might be connection or network issue. Is that linked server [XYZ] causing this? Not sure, i get it when i try to execute the last select statement that uses NOT EXISTS